We, the CloudShare Quality Assurance team, migrated from a web-based UI Automation implemented using C# and Watin (http://watin.org/) to a solution using C#, Selenium (http://docs.seleniumhq.org/) and Applitools Eyes (http://applitools.com/). With the use of Applitools Eyes, we validate all the UI elements of the application across multiple browsers (Google Chrome, Firefox and Internet Explorer) and across various resolutions.
Applitools Eyes has saved us quite a bit of time and effort from coding (including maintenance), and has allowed us to achieve much greater automated UI coverage. This tool has helped us find quite a few layout issues, flow issues as well as functional issue. As a result, we’ve reduced the overall number of missed bugs and improved the product quality.
We use CloudShare Production environments to host and run all our test environments (for testing both Production and our local Development labs). This includes machines from most modern Windows operating systems, and with all browsers combinations supported (Chrome, Firefox and Internet Explorer 9+). Our automation is executed with each deployment (which happens several times a day), with each Production deployment (every Sunday), and we have full regression cycles happening nightly and throughout the weekend.
Applitools Eyes allows labeling any regions to ignore (useful where we have animated gifs running as an example, or where we use dynamic data), or define floating regions (which can move several pixels depending on page), and these regions automatically propagate to all the screens that include similar elements. Additionally, when we make a change in the application which affects multiple pages in the same way (e.g. changing the header of our site), we only need to review and accept this change for a single window, and Applitools Eyes scans all other windows and automatically approves similar changes.
Following the addition of Applitools Eyes, the amount of Automation code has been greatly reduced (with almost all UI checks are now handled within the screenshot comparison) while increasing browser coverage, resolution coverage, overall test reliability and significantly lowering the time spent on code maintenance. With our previous solution, we only tested using Internet Explorer and missed a good number of bugs that only showed up on one of the other browsers. In addition, we now test with a minimum resolution of 1024*768 and up to 1920*1080 which lets us ensure the UI looks right.
Applitools provides constant functionality updates based on our feedback and was very simple to integrate into our Selenium based project. The latest releases provide OCR support (your can read text from regions labeled within their application and using your application screenshot). In addition, there will be support for several possible scenarios within a screen (when we have warnings on screen as an example during a deployment). As with Selenium, Applitools updates are handled via Nuget packages through Visual Studio.
Please note the following screenshots taken during automation execution and which demonstrate how UI bugs can be found using Applitools Eyes, and how you can configure dynamic data to be ignored when comparing screenshots.
Here’s a screenshot within Applitools that shows our Console access test, you can see the dynamic data regions that are ignored within the screen comparison:
(click image to get full size)
Here’s a screenshot within Applitools that shows our RDP access test, you can see the differences highlighted when the dynamic region that has the Windows time is not ignored:
(click image to get full size)
Here’s a screenshot within Applitools that shows a text overlap/layout issue that was found as part of using the Applitools screenshot comparison (with a resolution of 1024 by 768):
This is the story about how we reduced our UI Automation code by over 50% while greatly increasing the overall test coverage.