It's fall of 2018. PhantomJS is dead. We're using CSS Grid all over the place. And we cannot seem to get our visual regression testing tool, Wraith, to work with headless chrome inside a Docker container.
We're not a Ruby shop. Looking around at the tools available, we found none that worked the way we want them to and were current in any way -- it looks like development on Wraith has greatly slowed down. Meanwhile, it uses Selenium Webdriver to run screenshots in headless chrome, while Puppeteer has since arrived on the scene and seems far easier to work with. After stumbling across a great post describing how to visual regression testing using Puppeteer and Pixelmatch, we rolled up our sleeves and created Visualify as a drop-in replacement for Wraith in our CI system.
Visualify is a brand new visual regression testing tool, heavily inspired by Wraith, and supporting existing Wraith configuration files.
Visualify is up and running in production in our CI systems. It supports our basic needs to compare a list of paths on two copies of a site at multiple screen widths. We plan to expand this to work with historical snapshots taken by other tools, providing the image diff and gallery functionalities for other use cases. We also plan to add support for using CSS selectors to grab particular page elements, and possibly script support to hide elements using CSS selectors (so things like ads don't constantly break).
As it is, it improves upon Wraith by using headless chrome via Puppeteer, and it runs inside Docker (see example Dockerfile).
The easiest way to install and run Visualify is using Nix, which handles all dependencies automatically:
nix shell github:freelock/visualify
visualify --helpgit clone https://github.com/freelock/visualify.git
cd visualify
nix shell path:.
visualify --helpgit clone https://github.com/freelock/visualify.git
cd visualify
nix develop
# This gives you a shell with visualify and all dependencies availableThe Nix installation automatically:
- Installs Node.js and all npm dependencies
- Configures the Sharp image processing library
- Sets up browser detection for system Chrome/Chromium
- Works on Linux, macOS, and in containers
You must have a recent version of Node.js installed, with npm.
- Clone this repository:
git clone https://github.com/freelock/visualify.git - cd into the repository:
cd visualify npm installto install puppeteer, sharp, and the other dependencies.sudo npm linkto make visualify available in your $PATH.
Note: You'll also need Chrome or Chromium installed on your system for screenshot capture.
Visualify currently provides 4 separate operations, which are meant to run sequentially with the same arguments. Check out the capture.yaml file for the base configuration options -- this file comes straight from Wraith, and many options are not yet supported. It will use this file as defaults, unless otherwise specified.
In our CI system, we spread our configs across 2 yaml files -- a base "default" one, and a site-specific paths.yaml that contains a Yaml block with a "paths:" key. For example:
paths:
home: /
about: /about
contact: /contact
bogus: /bogus
sitemap: /sitemapIn addition, you also need a "domains" block to list the two sites you want to compare. Alternatively, you can pass these domains as arguments to visualify.
There are 4 phases/commands supported:
Capture a set of screenshots.
Example:
visualify capture -c paths.yaml -o shots dev http://devsite.dev prod https://mysite.prod
This creates a tree of directories in shots/, one for each path in the paths yaml key, plus another set under shots/thumbnails. Then it uses puppeteer to take a screenshot of each path on each domain at each screen_width.
Screenshots are taken serially to reduce load on our test server.
Compare two sets of screenshots using Pixelmatch.
Example:
visualify compare -c paths.yaml -o shots dev http://devsite.dev prod https://mysite.prod
This loops through all the screenshots specified by the configuration, resizes them to have the same height and width, and generates an image diff and a percent of pixels different.
Generate thumbnails of all images using sharp.
Example:
visualify thumbnail -c paths.yaml -o shots dev http://devsite.dev prod https://mysite.prod
This loops through all screenshots and diffs previously generated, and creates thumbnails at the size specified in the gallery config.
Generate an HTML gallery of the results.
Example:
visualify gallery -c paths.yaml -o shots dev http://devsite.dev prod https://mysite.prod
This gallery is taken straight out of Wraith, turned into a Mustache template.
Run all 4 steps in sequence: capture, compare, thumbnail, and gallery.
Example:
visualify all -c paths.yaml -o shots --debug dev http://devsite.dev prod https://mysite.prod
This command executes all phases sequentially, making it perfect for CI/CD pipelines or complete regression testing workflows. Use the --debug flag to see the browser during the capture phase.