* This is a contributed post from Chris Navrides, CEO at Dev Tools AI. The content is based on his presentation at ProvarLabs’ first internal Brown Bag session, a knowledge-sharing session featuring industry thought leaders, which reached 60+ attendees.

When writing user interface (UI) automation, the test is written for the state of the app as it was at that moment. After that moment, subsequent changes to the app or page will result in potentially broken or flaky tests. Changes can occur for a variety of reasons:

  • Refactors: Changing the class names to be more appropriate
  • Change of web frameworks: As an example, the asp.net app is being re-written to use react
  • Experimentation: Random experiments that can alter the UI or backend to change the user experience
  • Backend data changes: This can result in the UI test assumptions being challenged

These changes cause instability in the underlying UI tests that must be fixed and maintained. Luckily, artificial intelligence (AI) provides a solution for end-to-end testing of apps. Read on to learn more.

DOM and Flow Changes

The issues mentioned above are usually grouped into one of two types of errors: the under-page model/DOM (Document Object Model) changes or the flow (or test steps) the test needs to take to complete the desired scenario.

Dom changes

DOM changes occur when some value within the page is altered. These can be minor, such as a CSS class change, or significant, where entire elements disappear. When these happen in a given test case looking for them, it fails, even when the object is still visually there.

Flow changes

Flow changes occur when a different set of steps appear in the tested application. This is a common practice for things like seasonal promotions or alerting users. Many free apps randomly show users ads that can take over the entire screen. International companies also experience this where payments and security requirements of specific regions have additional demands. If a test is meant to execute 100% of the time, this presents a difficult challenge in all these scenarios.

AI Changes the Game

With AI and machine learning advances, computers can start to emulate human testers. Harnessing this technology can help solve the problems faced during test execution.

Element Selection

Building locators resistant to DOM changes within a UI testing framework can be achieved through visual AI to find the element on the screen. This has several advantages: it allows the test case to execute like an end user and not have false negatives when an underlying change occurs. It can also be executed across different platforms, such as iOS/Android and the web, as most applications use the same iconography for all users.

What this looks like in practice is a set of bounding boxes for a given screenshot (fig. 1) that the model detects. The model outputs boxes of what it sees are given elements (fig. 2).

Fig. 1: A reference image of www.nytimes.com 

Fig. 2: Element bounding boxes over the screenshot

Navigation

Tests can use reinforcement learning within a given application to assist in flow changes. This will allow them to robustly execute a given test without building the additional logic to check for possible new screens/pop-ups.

App Graph

To start with this approach, you must first understand the application. This can be accomplished by learning from existing tests or by doing a crawl. This aims to understand which buttons on a given screen result in going to a new screen.

You can determine which screen is which by using each screen’s URL or page name. Each element can be determined via its XPath or element name. Combining these will give you a graph where nodes represent a screen, and edges are elements.

Q-Learn

Upon building the app graph, you can apply a reinforcement learning approach. A popular reinforcement learning technique is QLearn. QLearn is an algorithm that seeks to find the best action to take, given the current state.

This state-action graph can then be applied where, for a given element you wish to interact with, you know which state/node it should be in. For each action in a given test script, a system can then look at the current state and the set of actions to get to where the element exists. It then changes the nature of a test script to only list what actions it should execute that are pertinent to the core logic being tested by that test case.

In Conclusion

AI and machine learning are being used for testing today. They have real applications at companies, both big and small. Various methods can be used, and various areas can be applied. In the future, all of these ways will be common in test frameworks, so those looking to stay ahead of the curve should consider implementing them now.

To learn more about Dev Tools AI, visit www.dev-tools.ai. For more on ProvarLabs’ current initiatives, which include these Brown Bag sessions and much more, visit provar.com/provarlabs.