Automated Visible Regression Testing With Playwright

Evaluating visible artifacts is usually a highly effective, if fickle, strategy to automated testing. Playwright makes this appear easy for web sites, however the particulars may take a bit finessing.

Current downtime prompted me to scratch an itch that had been plaguing me for some time: The model sheet of a web site I preserve has grown just a bit unwieldy as we’ve been including code whereas exploring new options. Now that we’ve got a greater thought of the necessities, it’s time for inner CSS refactoring to pay down a few of our technical debt, profiting from fashionable CSS options (like utilizing CSS nesting for extra apparent construction). Extra importantly, a cleaner basis ought to make it simpler to introduce that darkish mode function we’re sorely missing so we will lastly respect customers’ most well-liked coloration scheme.

Nevertheless, being of the apprehensive persuasion, I used to be reluctant to make massive adjustments for concern of unwittingly introducing bugs. I wanted one thing to protect in opposition to visible regressions whereas refactoring — besides which means snapshot testing, which is notoriously sluggish and brittle.

On this context, snapshot testing means taking screenshots to determine a dependable baseline in opposition to which we will evaluate future outcomes. As we’ll see, these artifacts are influenced by a large number of things that may not at all times be totally controllable (e.g. timing, variable {hardware} assets, or randomized content material). We even have to take care of state between take a look at runs, i.e. save these screenshots, which complicates the setup and means our take a look at code alone doesn’t totally describe expectations.

Having procrastinated with out a extra agreeable resolution revealing itself, I lastly got down to create what I assumed could be a fast spike. In any case, this wouldn’t be a part of the common take a look at suite; only a one-off utility for this specific refactoring job.

Happily, I had obscure recollections of previous analysis and shortly rediscovered Playwright’s built-in visible comparability function. As a result of I attempt to choose dependencies rigorously, I used to be glad to see that Playwright appears to not depend on many exterior packages.

Setup

The really helpful setup with npm init playwright@newest does an honest job, however my minimalist style had me set every part up from scratch as a substitute. This do-it-yourself strategy additionally helped me perceive how the totally different items match collectively.

Provided that I anticipate snapshot testing to solely be used on uncommon events, I needed to isolate every part in a devoted subdirectory, known as take a look at/visible; that will likely be our working listing from right here on out. We’ll begin with package deal.json to declare our dependencies, including a couple of helper scripts (spoiler!) whereas we’re at it:

{
  "scripts":  true"
  ,
  "devDependencies": {
    "@playwright/take a look at": "^1.49.1"
  }
}

If you happen to don’t need node_modules hidden in some subdirectory but in addition don’t wish to burden the foundation undertaking with this rarely-used dependency, you may resort to manually invoking npm set up --no-save @playwright/take a look at within the root listing when wanted.

With that in place, npm set up downloads Playwright. Afterwards, npx playwright set up downloads a variety of headless browsers. (We’ll use npm right here, however you may favor a distinct package deal supervisor and job runner.)

We outline our take a look at atmosphere through playwright.config.js with a couple of dozen primary Playwright settings:

import { defineConfig, gadgets } from "@playwright/take a look at";

let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
let BASE_URL = "http://localhost:8000";
let SERVER = "cd ../../dist && python3 -m http.server";

let IS_CI = !!course of.env.CI;

export default defineConfig({
  testDir: "./",
  fullyParallel: true,
  forbidOnly: IS_CI,
  retries: 2,
  staff: IS_CI ? 1 : undefined,
  reporter: "html",
  webServer: {
    command: SERVER,
    url: BASE_URL,
    reuseExistingServer: !IS_CI
  },
  use: {
    baseURL: BASE_URL,
    hint: "on-first-retry"
  },
  initiatives: BROWSERS.map(ua => ({
    identify: ua.toLowerCase().replaceAll(" ", "-"),
    use: { ...gadgets[ua] }
  }))
});

Right here we anticipate our static web site to already reside throughout the root listing’s dist folder and to be served at localhost:8000 (see SERVER; I favor Python there as a result of it’s broadly obtainable). I’ve included a number of browsers for illustration functions. Nonetheless, we’d cut back that quantity to hurry issues up (thus our easy BROWSERS checklist, which we then map to Playwright’s extra elaborate initiatives information construction). Equally, steady integration is YAGNI for my specific situation, in order that entire IS_CI dance may very well be discarded.

Seize and evaluate

Let’s flip to the precise checks, beginning with a minimal pattern.take a look at.js file:

import { take a look at, anticipate } from "@playwright/take a look at";

take a look at("house web page", async ({ web page }) => {
  await web page.goto("https://css-tricks.com/");
  await anticipate(web page).toHaveScreenshot();
});

npm take a look at executes this little take a look at suite (based mostly on file-name conventions). The preliminary run at all times fails as a result of it first must create baseline snapshots in opposition to which subsequent runs evaluate their outcomes. Invoking npm take a look at as soon as extra ought to report a passing take a look at.

Altering our website, e.g. by recklessly messing with construct artifacts in dist, ought to make the take a look at fail once more. Such failures will provide varied choices to match anticipated and precise visuals:

We are able to additionally examine these baseline snapshots immediately: Playwright creates a folder for screenshots named after the take a look at file (pattern.take a look at.js-snapshots on this case), with file names derived from the respective take a look at’s title (e.g. home-page-desktop-firefox.png).

Producing checks

Getting again to our unique motivation, what we wish is a take a look at for each web page. As an alternative of arduously writing and sustaining repetitive checks, we’ll create a easy internet crawler for our web site and have checks generated robotically; one for every URL we’ve recognized.

Playwright’s international setup allows us to carry out preparatory work earlier than take a look at discovery begins: Decide these URLs and write them to a file. Afterward, we will dynamically generate our checks at runtime.

Whereas there are different methods to move information between the setup and test-discovery phases, having a file on disk makes it simple to change the checklist of URLs earlier than take a look at runs (e.g. briefly ignoring irrelevant pages).

Web site map

Step one is to increase playwright.config.js by inserting globalSetup and exporting two of our configuration values:

export let BROWSERS = ["Desktop Firefox", "Desktop Chrome", "Desktop Safari"];
export let BASE_URL = "http://localhost:8000";

// and so on.

export default defineConfig({
  // and so on.
  globalSetup: require.resolve("./setup.js")
});

Though we’re utilizing ES modules right here, we will nonetheless depend on CommonJS-specific APIs like require.resolve and __dirname. It seems there’s some Babel transpilation taking place within the background, so what’s really being executed might be CommonJS? Such nuances generally confuse me as a result of it isn’t at all times apparent what’s being executed the place.

We are able to now reuse these exported values inside a newly created setup.js, which spins up a headless browser to crawl our website (simply because that’s simpler right here than utilizing a separate HTML parser):

import { BASE_URL, BROWSERS } from "./playwright.config.js";
import { createSiteMap, readSiteMap } from "./sitemap.js";
import playwright from "@playwright/take a look at";

export default async operate globalSetup(config) {
  // solely create website map if it does not exist already
  strive {
    readSiteMap();
    return;
  } catch(err) {}

  // launch browser and provoke crawler
  let browser = playwright.gadgets[BROWSERS[0]].defaultBrowserType;
  browser = await playwright[browser].launch();
  let web page = await browser.newPage();
  await createSiteMap(BASE_URL, web page);
  await browser.shut();
}

That is pretty boring glue code; the precise crawling is going on inside sitemap.js:

• createSiteMap determines URLs and writes them to disk.
• readSiteMap merely reads any beforehand created website map from disk. This will likely be our basis for dynamically producing checks. (We’ll see later why this must be synchronous.)

Happily, the web site in query gives a complete index of all pages, so my crawler solely wants to gather distinctive native URLs from that index web page:

operate extractLocalLinks(baseURL) {
  let urls = new Set();
  let offset = baseURL.size;
  for(let { href } of doc.hyperlinks) {
    if(href.startsWith(baseURL)) {
      let path = href.slice(offset);
      urls.add(path);
    }
  }
  return Array.from(urls);
}

Wrapping that in a extra boring glue code provides us our sitemap.js:

import { readFileSync, writeFileSync } from "node:fs";
import { be a part of } from "node:path";

let ENTRY_POINT = "/subjects";
let SITEMAP = be a part of(__dirname, "./sitemap.json");

export async operate createSiteMap(baseURL, web page) {
  await web page.goto(baseURL + ENTRY_POINT);
  let urls = await web page.consider(extractLocalLinks, baseURL);
  let information = JSON.stringify(urls, null, 4);
  writeFileSync(SITEMAP, information, { encoding: "utf-8" });
}

export operate readSiteMap() {
  strive {
    var information = readFileSync(SITEMAP, { encoding: "utf-8" });
  } catch(err) {
    if(err.code === "ENOENT") {
      throw new Error("lacking website map");
    }
    throw err;
  }
  return JSON.parse(information);
}

operate extractLocalLinks(baseURL) {
  // and so on.
}

The attention-grabbing bit right here is that extractLocalLinks is evaluated throughout the browser context — thus we will depend on DOM APIs, notably doc.hyperlinks — whereas the remainder is executed throughout the Playwright atmosphere (i.e. Node).

Checks

Now that we’ve got our checklist of URLs, we principally simply want a take a look at file with a easy loop to dynamically generate corresponding checks:

for(let url of readSiteMap()) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    await web page.goto(url);
    await anticipate(web page).toHaveScreenshot();
  });
}

Because of this readSiteMap needed to be synchronous above: Playwright doesn’t at the moment assist top-level await inside take a look at recordsdata.

In follow, we’ll need higher error reporting for when the positioning map doesn’t exist but. Let’s name our precise take a look at file viz.take a look at.js:

import { readSiteMap } from "./sitemap.js";
import { take a look at, anticipate } from "@playwright/take a look at";

let sitemap = [];
strive {
  sitemap = readSiteMap();
} catch(err) {
  take a look at("website map", ({ web page }) => {
    throw new Error("lacking website map");
  });
}

for(let url of sitemap) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    await web page.goto(url);
    await anticipate(web page).toHaveScreenshot();
  });
}

Getting right here was a little bit of a journey, however we’re just about completed… except we’ve got to cope with actuality, which usually takes a bit extra tweaking.

Exceptions

As a result of visible testing is inherently flaky, we generally have to compensate through particular casing. Playwright lets us inject customized CSS, which is usually the simplest and simplest strategy. Tweaking viz.take a look at.js…

// and so on.
import { be a part of } from "node:path";

let OPTIONS = {
  stylePath: be a part of(__dirname, "./viz.tweaks.css")
};

// and so on.
  await anticipate(web page).toHaveScreenshot(OPTIONS);
// and so on.

… permits us to outline exceptions in viz.tweaks.css:

/* suppress state */
fundamental a:visited {
  coloration: var(--color-link);
}

/* suppress randomness */
iframe[src$="/articles/signals-reactivity/demo.html"] {
  visibility: hidden;
}

/* suppress flakiness */
physique:has(h1 a[href="https://css-tricks.com/wip/unicode-symbols/"]) {
  fundamental tbody > tr:last-child > td:first-child {
    font-size: 0;
    visibility: hidden;
  }
}

:has() strikes once more!

Web page vs. viewport

At this level, every part appeared hunky-dory to me, till I spotted that my checks didn’t really fail after I had modified some styling. That’s not good! What I hadn’t taken into consideration is that .toHaveScreenshot solely captures the viewport moderately than the whole web page. We are able to rectify that by additional extending playwright.config.js.

export let WIDTH = 800;
export let HEIGHT = WIDTH;

// and so on.

  initiatives: BROWSERS.map(ua => ({
    identify: ua.toLowerCase().replaceAll(" ", "-"),
    use: {
      ...gadgets[ua],
      viewport: {
        width: WIDTH,
        peak: HEIGHT
      }
    }
  }))

…after which by adjusting viz.take a look at.js‘s test-generating loop:

import { WIDTH, HEIGHT } from "./playwright.config.js";

// and so on.

for(let url of sitemap) {
  take a look at(`web page at ${url}`, async ({ web page }) => {
    checkSnapshot(url, web page);
  });
}

async operate checkSnapshot(url, web page) {
  // decide web page peak with default viewport
  await web page.setViewportSize({
    width: WIDTH,
    peak: HEIGHT
  });
  await web page.goto(url);
  await web page.waitForLoadState("networkidle");
  let peak = await web page.consider(getFullHeight);

  // resize viewport for earlier than snapshotting
  await web page.setViewportSize({
    width: WIDTH,
    peak: Math.ceil(peak)
  });
  await web page.waitForLoadState("networkidle");
  await anticipate(web page).toHaveScreenshot(OPTIONS);
}

operate getFullHeight() {
  return doc.documentElement.getBoundingClientRect().peak;
}

Notice that we’ve additionally launched a ready situation, holding till there’s no community site visitors for some time in a crude try to account for stuff like lazy-loading photographs.

Bear in mind that capturing the whole web page is extra resource-intensive and doesn’t at all times work reliably: You might need to cope with structure shifts or run into timeouts for lengthy or asset-heavy pages. In different phrases: This dangers exacerbating flakiness.

Conclusion

A lot for that fast spike. Whereas it took extra effort than anticipated (I imagine that’s known as “software program growth”), this may really clear up my unique downside now (not a standard function of software program as of late). In fact, shaving this yak nonetheless leaves me itchy, as I’ve but to do the precise work of scratching CSS with out breaking something. Then comes the true problem: Retrofitting darkish mode to an present web site. I simply may want extra downtime.