* This is a post contributed by Chris Navrides, CEO at Dev Tools AI. The content is based on his presentation at ProvarLabs’ first internal Brown Bag session, a knowledge-sharing session featuring industry thought leaders, which reached 60+ attendees.

When writing user interface (UI) automation, you write the test for the app’s state as it was at that moment. After that moment, subsequent changes to the app or page will result in potentially broken or flaky tests. Changes can occur for a variety of reasons:

  • Refactors: Changing the class names to be more appropriate
  • Change of web frameworks: As an example, the asp.net app is being re-written to use react
  • Experimentation: Random experiments that can alter the UI or backend to change the user experience
  • Backend data changes: This can result in the UI test assumptions being challenged

These changes create instability in the underlying UI tests, requiring fixes and maintenance. Luckily, artificial intelligence (AI) provides a solution for end-to-end app testing. Read on to learn more.

DOM and Flow Changes

People usually group the mentioned issues into one of two types of errors: the under-page model/DOM (Document Object Model) changes or the flow (or test steps) the test needs to take to complete the desired scenario.

Dom changes

DOM changes happen when someone alters a value within the page. These can be minor, such as a CSS class change, or significant, where entire elements disappear. When these happen in a given test case looking for them, it fails, even when the object is still visually there.

Flow changes

Flow changes occur when a different set of steps appear in the tested application. This is a common practice for things like seasonal promotions or alerting users. Many free apps randomly show users ads that can take over the entire screen. International companies also experience this where payments and security requirements of specific regions have additional demands. If a test is meant to execute 100% of the time, all these scenarios present a difficult challenge.

AI Changes the Game

With AI and machine learning advances, computers can start to emulate human testers. Harnessing this technology can help solve the problems faced during test execution.

Element Selection


Using visual AI to find the element on the screen can help you achieve building locators that are resistant to DOM changes within a UI testing framework.

This has several advantages: it allows the test case to execute like an end user and not have false negatives when an underlying change occurs. Most applications use the same iconography for all users, allowing it to be executed across different platforms, such as iOS/Android and the web.

In practice, this looks like a set of bounding boxes for a given screenshot (fig. 1) that the model detects. The model outputs boxes of what it sees as given elements (fig. 2).

Fig. 1: A reference image of www.nytimes.com 

Fig. 2: Element bounding boxes over the screenshot


Tests can use reinforcement learning within a given application to facilitate flow changes. This will allow them to robustly execute a given test without building the additional logic to check for possible new screens/pop-ups.

App Graph

To start with this approach, you must first understand the application. This can be accomplished by learning from existing tests or by doing a crawl. This aims to understand which buttons on a given screen result in going to a new screen.

You can determine which screen is which by using each screen’s URL or page name. Each element can be determined via its XPath or element name. Combining these will give you a graph where nodes represent a screen and edges are elements.


Upon building the app graph, you can apply a reinforcement learning approach. A popular reinforcement learning technique is QLearn. QLearn is an algorithm that seeks to find the best action to take, given the current state.

This state-action graph can then be applied where, for a given element you wish to interact with, you know which state/node it should be in. For each action in a given test script, a system can look at the current state and the set of actions to get to where the element exists.

It then transforms the nature of a test script to list only the actions it should execute pertinent to the core logic being tested by that test case.


In Conclusion

AI and machine learning are being used for testing today. They have real applications at companies, both big and small. Various methods and areas can be applied. In the future, all of these methods will be common in test frameworks, so those looking to stay ahead of the curve should consider implementing them now.

To learn more about Dev Tools AI, visit www.dev-tools.ai. For more on ProvarLabs’ current initiatives, which include these Brown Bag sessions and much more, visit provar.com/provarlabs.