Re: Musings on Automated UI Testing: Our Experiences with Ghost Inspector

Azucena Zuora Staff

Zuora Staff

Musings on Automated UI Testing: Our Experiences with Ghost Inspector

by Zuora Staff on ‎01-09-2018 11:26 AM

Introduction


As Zuora continues to Reinvent, implementing new backend architectures that allow us to move from a monolith to microservices, we’re also introducing changes that revamp our product’s UI. This is outlined in Wei Liu’s recent blog post.


Our Platform UI Team is actively working towards enhancing the user experience (UX), and have been rolling out granular changes including pixel adjustments, new icons, and word modifications that can introduce subtle or sometimes significant shifts in UI layout, icon disparity, and typography.


Due to the expansiveness of the changes being rolled out, verifying the correctness of these changes can not be covered by manual testing alone; verification needs to be augmented with UI automation that has the ability to refer to a source of truth every time there are UI changes. The goal being to decrease the amount of manual testing hours after each deployment and, ultimately, the need for manual testing, with the hopes of catching UI regressions more quickly and effectively.

 

Quick Intro to Ghost Inspector


Ghost Inspector is an automation testing tool designed to cover full front-end UI testing. It comes with a variety of useful features that help QA Engineers cover almost every use case, including:

 

  • Pixel-by-pixel comparison
  • Test run scheduling
  • Multi-browser support for test suites
  • Multi-geo support for tests
  • Chrome plugin to quickly record new test steps
  • Jenkins integration
  • AWS Console integration
  • Bitbucket/Github integration
  • Mobile testing
  • Test export capabilities: Selenium JSON or HTML formats
  • Ability to add assertions while recording test steps

 

How we use Ghost Inspector


There are a few Zuora product teams that use Ghost Inspector today:

 

Analytics

 

Zuora’s Analytics releases are very heavy in new features and UI enhancements. This makes it difficult to keep up with every change that is introduced - be it via new code or library upgrades. Ghost Inspector’s pixel-by-pixel comparison is extremely useful in supporting large feature deltas between releases. Also, daily scheduled test runs drop the team a message via slack/email whenever a test case fails.

 

Figure 1

 

Figure 1 - Zuora Insights UI

 

Zuora for Salesforce


The Zuora for Salesforce team is currently evaluating Ghost Inspector as well. The primary use cases are to test the interaction between Salesforce and Zuora. In particular, Ghost Inspector enables the team to do two things:

 

  • Bridge the gap that exists between our current manual tests and our existing automated tests
  • Quickly spin up new modular test cases. This enables the team to leverage the full breadth of features that Ghost Inspector provides 

Here is what test steps look like in Ghost Inspector:

 

Figure 2

 Figure 2 - Ghost Inspector Test Cases

 

Note that attributes (e.g., #id_username) can be used to locate elements in the product UI.


How Ghost Inspector Stacks Up

 

Ghost Inspector has been compared to a standard Selenium and webdriver approach. After comparing Ghost Inspector to a standard Selenium and webdriver approach, we discovered that Ghost Inspector surpasses Selenium and can benefit developers in ease of use, test readability, and debugging capability.


Based on our experience, we’ve identified the following pros and cons of using Ghost Inspector:

 

Pros Cons
Good variety of UI actions such as drag and drop, mouse over, keypress etc. Unable to loop through multiple UI elements
Ability to select timeout for each test Recording doesn’t always recognize elements
Modularize test steps XPaths are not optimized
Make assertions while recording Relies more on XPath rather than CSS
Set global variables Tests can fail due to selectors not having a unique ID or attribute identifier, as well as the recorder recording unreliable XPaths
Low monthly cost approx. $500/month
Use spreadsheet to input test cases
Easy to ramp up on software
Flexibility to run test suite in sandbox, production, and staging environments
No need to download code locally to be up and running on suite

 

What’s next?


The Analytics team will use Ghost Inspector more broadly while enhancing the test cases to reduce brittleness and increase maintainability.


The Zuora for Salesforce team is still trialling Ghost Inspector and will make a decision about how Ghost Inspector could fit into their long-term QA strategy.

Comments
AmyLy Zuora Moderator

Zuora Moderator

by Zuora Moderator
on ‎01-09-2018 07:53 PM

Thanks for the post, @Azucena! Super interesting read, especially as someone who isn't familiar with any QA processes at all.