One of the significant challenges of Salesforce test sandbox automation is keeping up with the changes that Salesforce delivers. In each significant release (Spring, Summer, and Winter) and any other changes that get delivered in between. Provar spends a lot of testing and engineering time to ensure that we deliver a Provar release where:
- Provar is compatible with every new Salesforce release
- Is available when you start testing with the Salesforce sandbox preview
- Lets you focus on testing new functionality and not updating existing tests to work with the latest Salesforce changes.
Identifying Two Key Problems With Salesforce Releases.
Problem 1: Test Tooling/Platform Mismatch
- Salesforce manages and evolves the page renderer for a Salesforce page. The developer doesn’t control how the source is rendered.
- Salesforce regularly changes how a page is rendered. It directly affects the HTML, CSS, Javascript, and DOM that most test automation tools rely on to build tests.
Here’s the problem: most Salesforce test automation tools use the same approach to testing Salesforce. They use it to test any web application. Specifically, those tools map page elements and build tests based on information about the rendered page: the page’s HTML, CSS, JavaScript, and DOM. While this works exceptionally well for most web applications, it’s a problem for Salesforce testing for two key reasons:
Here’s a simplified picture of how Salesforce renders a web page.
Diagram of Salesforce Architecture
Starting from the bottom of the diagram:
- A Salesforce page is built around metadata (Page layout definitions, objects, and field definitions). It has business data (account names, opportunity close dates, mailing addresses, etc.).
- A page renderer (Classic, Aura, or LWC) processes the business data and metadata. It creates the web page source.
- Interpreting the web page source by a web browser, displaying the results on a screen.
Salesforce controls the bottom two layers by defining the form and structure of the page. Consequently, it also determines how the page source code is rendered.
In-house and deployed enterprise web applications allow developers to control the source code rendering. Consequently, developers can define elements (such as unique IDs), which makes setting up test automation much more accessible.
As Salesforce controls and changes how the page is rendered, developers cannot influence factors that could make a test more resilient to changes. Consequently, tests built on the rendered source (the presentation layer) are more prone to breaking with Salesforce updates. Therefore, we’ve provided a summary of some of the critical differences between Salesforce, enterprise, and in-house web app development that affect testing:
Problem 2: Strict Time Limits for Compatibility Testing
The other important piece that the Salesforce developers don’t control is how long they must deliver compatibility with each new update. The sandboxed preview is most customers’ first chance to check out the newest release. Previews last from 3 – 4 weeks, and then it’s GA. Customers have those weeks to:
- Run their typical regression test to identify potential issues with their customizations (including applications)
- Identify the tests that break
- Determine if it’s a mapping problem (because of a Salesforce change)
- Fix the mapping and rerun the tests
- Identify remaining errors as customization bugs
- Fix the bugs and rerun the tests
The size of the regression test grows along with the amount of customization, making it more and more challenging to fit compatibility testing into the sandbox preview window. Eventually, this builds into what Provar calls the rework spiral. This happens when companies spend more time maintaining test compatibility than adding test automation to ensure the quality of new features and applications. The cycle repeats with every release, and the problem worsens as the size of the regression test suite increases. As a result, clients quickly choose between boosting testing resources to stay up or accepting decreased coverage and increased risk.
The graphic below shows the typical Salesforce release schedule. Partners get an early release 4 to 5 weeks before a sandbox preview, the first time customers can begin testing the release. GA happens quickly after the sandbox preview, giving customers a limited time to fix their tests and identify errors.
Pro-Tip #1: Plan Ahead – The rework spiral problem becomes apparent only after an organization realizes that manual testing cannot address all its needs and decides to transition to automated testing. Consequently, dealing with a growing number of breaking tests often goes unnoticed for several Salesforce release cycles. It takes time to a) flag this issue as a regular occurrence and b) gather enough tests to recognize it as a problem. By that point, the team will have already invested a significant amount of time, money, and effort into building the test automation. Most of the work isn’t transferable to another platform. So, plan your automation well and choose your tools carefully.
How Provar Delivers Salesforce Release Compatibility and Unbreakable Tests
Solution Step 1: Testing based on the Salesforce metadata model
Provar was purpose-built for Salesforce, with critical design criteria to address the test tooling/platform mismatch. Instead of using the generated web page source to create a test, Provar uses information from the Salesforce metadata model and metadata (including modifications) from the tested org. Here’s a graphic showing how Provar works with Salesforce.
Provar Test Automation Diagram
Building a Salesforce test in Provar involves:
- The developers explicitly engineered Provar for Salesforce with crucial design criteria to resolve the test tooling/platform mismatch.
- Caching the metadata in Provar and organizing the related test step data (unique Salesforce field identifiers, field interactions, and data values)
- Executing Provar’s version of a page renderer – the Provar Salesforce layout engine – to automatically create a fully configured test step for any Salesforce page element (as simple as a single mouse click to add a test step)
How is this different from other solutions?
- Building a test in Provar does not rely on the rendered HTML, CSS, or Javascript.
- Page layout definitions, object and field definitions, interactions, and Salesforce field identifiers all exist below the web page source level and change much less frequently. (Salesforce needs these elements to be stable or risk breaking existing applications and customizations.)
- Provar maintains all the code used to map Salesforce page elements, predict interactions, understand page layouts, and automatically generate and execute test steps in Provar. With other web application testing tools, the tester maintains all the code defining web page mapping and test step creation.
Solution Step 2: Maintaining Salesforce Release Compatibility
Provar gets access to Salesforce releases 4 – 5 weeks before Sandbox previews as part of the Salesforce Monorail program. During this time, Provar runs approximately 2,500 Salesforce test sandboxes to identify where Salesforce has made changes that could potentially affect Provar’s test-building applications. The changes can be significant (such as shifting from Classic to Lightning) or small (like moving anchors on a Sales console page from Aura to LWC). Therefore, after evaluating these changes and the results of the regression tests, Provar updated its application to handle them. Consequently, we added Provar’s compatibility and Provar customer testing to the same diagram.
The net effect: Provar maintains all of the code related to creating a test in Salesforce so that:
- Provar customers don’t need to check the salesforce test sandbox for release compatibility
- Test failures identified in a sandbox preview are more likely to be actual bugs (vs. compatibility bugs)
- You can avoid rework spirals
- Companies can confidently meet Salesforce GA deadlines
Pro-Tip #2: Analyze other tools for compatibility – Do other Salesforce test sandbox tool vendors get access via Monorail, and can they provide compatibility? Yes, other vendors are part of the Monorail program. However, the test builder creates the test automation code and, more importantly, the web page element mapping. Since individual test builders often create different web locators for the same element, the customer must maintain all of that mapping code. So, the vendor has no way of providing release-to-release test compatibility.
Want to find out how to get out of the rework spiral? Schedule a demo, and let’s talk about how to make moving to a new Salesforce release hassle-free.
