About the Company
More than a century ago, this financial company was established as a life insurance company with its roots centered around the pillars of dependability, honesty and service. Building on that solid foundation, it has since grown to become a global leader in the financial services industry serving more than 17 million customers worldwide and is currently a Fortune 500 company offering a diverse range of financial services and solutions.
With a strong focus on four core business areas — life insurance, annuities, retirement planning services and group protection — the business is built around supporting, preserving and enhancing customers’ lifestyles and providing better retirement outcomes.
- Established in 1905
- Led by over 9,000 employees
- Listed in the top 180 on the 2021 Fortune 500 list by revenue, top 25 by assets
- Industry: Financial services
- Salesforce orgs: 5 orgs. The largest instance has 100 profiles, 1,300 users and 46 unique user/org combinations
- Salesforce users: 2,000 (1,300 users in the largest instance)
- QA team size: 5 Provar licenses
- Provar tests: 4,000 in total (1,600 tests in the largest instance)
- Environments tested with Provar: QA sandbox, development sandbox and a smoke test environment
The Importance of Salesforce
Like many Fortune 500 companies, this company relies heavily on Salesforce as a crucial part of their success. While each org is customized to support varying lines of business, the primary use case for the largest org is to support transactions for internal and external wholesalers who sell policies for the company’s advisors. There are also other Salesforce instances used by retirement consultants who evaluate retirement planning information for customers and consult with plan sponsors. The company also leverages one instance that is dedicated to group protection which handles the legal protection requirements.
“Keeping our Salesforce orgs running as expected is absolutely critical to the success of our company and our customers’ financial well-being around the globe. It’s not something we take lightly.”
– Quality Assurance Analyst
While the platform was originally designed to support sales operations alone, it has since expanded to facilitate a wide range of activities enterprise-wide. At the company, Salesforce is used to optimize the customer journey and support customer service, marketing automation, analytics processes and more.
Today at the company, 2,000+ team members currently leverage Salesforce to collaborate each day to accelerate processes, support customer requests, document quotes and track opportunities – spanning functions including finance, sales, marketing and operations.
The Risks of Broken Functionality
While the company was benefitting from the unsurpassed ease of use of Salesforce (e.g. the ability to easily customize the application and streamline internal operations), in 2017 they realized that manual testing alone was not a sufficient approach for testing such a complex application for a variety of reasons.
The risk of deploying new releases with broken functionality could have severe consequences for the company. Most notably, there are legal risks if broken processes impact the delivery of financial services outlined in contracts. In addition, software defects could also reduce employee productivity, prolong sales cycles and increase customer churn.
In addition to testing customizations and new in-house applications, three major Salesforce releases each year had the potential to disrupt existing functionality within their orgs. Since the company was supporting more than 17 million customers worldwide, they needed a better way to mitigate risk and protect their investment in Salesforce.
Manual Testing Challenges
Until 2017, the QA team had been relying on manual testing to find and fix bugs, though due to the complex nature of their business and their use of Salesforce, manual testing alone was introducing too much risk.
Challenge #1: The Tremendous Scope of Work
To support the immense scope of their global operations, the company uses 5 Salesforce org instances which are used by 2,000+ employees. Compounding that level of complexity, team members use Salesforce in a variety of browsers and there are varying levels of permissions. For example, the largest Salesforce instance has 100 profiles, 1,300 users and 46 Salesforce connections. And, the QA team also regularly needed to conduct tests in three environments: a QA sandbox, a dev sandbox and a smoke check sandbox.
So what does that mean for testing complexity? If the team needs to test 100 new features, for example, that could easily escalate into tens of thousands of test case combinations which would not be feasible even with a large QA team. If you calculate the sheer volume of possible combinations of scenarios that need to be tested, it is virtually impossible for their small team to get adequate testing coverage with manual testing alone.
“When we had to rely on manual testing, we weren’t able to test all of the impacted profiles. For example, if a change impacted 10 profiles, we didn’t have the bandwidth to test all of them. We had to prioritize the most important ones and only test those which really put us at risk. With Provar, when one test case is created, we can just copy that test case and adjust the variables to test all of them.”
– Quality Assurance Analyst
Challenge #2: Heavily Customized Salesforce Orgs
In addition to testing changes introduced by Salesforce with major releases, the company frequently adds new customized features and functionality to each of their five Salesforce orgs.
This added another layer of complexity and pressure for the QA team since the development team had aggressive release schedules for each org. While the team needed to adequately test each customization prior to deployment, they had to keep up with the pace of change to avoid delaying releases.
Challenge #3: Unknown Dependencies
While the ability to easily make changes to the application is one of the key benefits afforded to Salesforce customers, it does introduce risk. Specifically, without accounting for all of an org’s dependencies, making changes in one workflow can inadvertently impact other workflows. With manual testing, users are only testing what they think may change and this doesn’t include the full spectrum of changes that may be introduced throughout the entire org.
Challenge #4: Design Complexity
With the rapid adoption of Aura and Lightning Web Components (and complex components such as dynamic forms, accordions, etc.), relying on manual testing alone limited the team’s ability to accurately test the scope of vastly complex features.
Challenge #5: Time-Consuming Manual Work
Given the relatively small size of the team and the time available for test execution, the company was limited to 400 manual tests per month.
The First Attempt
Given the increasing importance of Salesforce within the organization and the limitations of manual testing, the team temporarily transitioned from manual testing to Selenium test automation but with limited success. They initially built out 400 tests, though changes introduced both internally by the team and externally by Salesforce would often break existing regression tests and require intensive test maintenance.
After spending months establishing a library of Selenium tests, the team was forced to spend weeks updating those existing tests after each major release.
“After switching from manual testing to Selenium, we quickly realized that it wasn’t sustainable. We had to put a lot of time into test maintenance after the first major Salesforce release and quickly realized that we needed to change. With Provar, tests keep working release after release.”
– Quality Assurance Lead
It also limited their ability to get the testing coverage that they needed since Selenium requires coding experience, a skill that only a portion of their team had.
Given the maintenance overhead and the need to expand the team’s coding experience to use Selenium, the team realized that they would not be able to meet their text coverage goals.
The Switch to Provar Automation
“Provar Automation stood out from the beginning. The easy click-to-build method to create tests empowers non-coders and coders alike. Anybody on our team should be able to build a test case regardless of their experience level. That was the primary reason why we chose Provar Automation.”
– Quality Assurance Lead
After its brief experiment with Selenium, the company sought to find a new solution and began a vendor evaluation process to research test automation solutions. After seeing a demo at Dreamforce, the company worked with Provar’s pre-sales team for a tailored assessment of their needs.
After evaluating potential options, the team selected Provar Automation for two primary reasons. Firstly, the substantial reduction in test maintenance given Provar Automation’s ability to work with Salesforce’s underlying data model and the ease-of-use of a low-code solution.
Ending the Rework Spiral with Unbreakable Salesforce Tests
One reason why we chose Provar Automation is that tests are future-proofed since you work with the underlying data model of Salesforce. It frees us from a ridiculous amount of test maintenance we would have to do with other tools like Selenium. Tests are more reusable and resilient to change.”
– Quality Assurance Lead
One of the most common challenges that QA teams face when using test automation for Salesforce is that tests often break. While this can happen for a variety of reasons, new Salesforce releases are the most common trigger for breaking automated tests because they can introduce a wide variety of changes including new user interface frameworks (Visualforce, Aura, and Lightning Web Components) and rendering changes with HTML, Javascript, and CSS modifications. Additionally, new versions may include layout updates and new API versions.
While these changes are designed to improve the user experience, platform performance and usability, these updates often cause problems for QA teams that use test frameworks that are more static in nature, similar to Selenium. Why is that? When hard-coded tests are created in one environment (where each and every element interaction is precisely defined), tests will fail when they are run in an updated environment that looks and functions differently. The result? Teams often end up in an endless rework spiral spending weeks fixing existing tests instead of scaling their automation portfolio.
In contrast, Provar Automation tests are much more resilient because Provar Automation is designed to work with Salesforce since tests are tied to the underlying data model. Using Provar Automation’s Test Builder, you can simply click on a field or button and the solution will automatically add all of the necessary information to your test.
What makes Provar Automation unique is that when you run the test at a later date, the solution remembers the Salesforce environment in which the test was created, understands how Salesforce has changed since then and adapts accordingly.
The result? Tests that keep working.
Low-Code Testing
Another feature that stood out for the company when evaluating vendors was Provar Automation’s Test Builder, an intuitive solution that allows testers to build test cases. With Test Builder, you can build tests in a natural and streamlined way, authoring and debugging as you go. With an intuitive and simple design that delivers fully configured test steps in real time, testers without a background in coding can start building test cases in minutes.
“Scripting in Provar Automation is extremely fast with the simple, point-and-click approach.”
– Quality Assurance Analyst
How the Company Uses Provar Automation
After completing Provar’s Jumpstart Implementation training, the team has been able to build a vast library of 4,000 tests (in comparison to only 400 with Selenium) that focus on functional regression and health checks. This is also impressive given that their relatively small team of ten has been able to accomplish this in less than two years.
Provar Automation has enabled the company to implement in-sprint automation and has integrated Provar Automation with GitLab to help their team find and fix bugs quickly and accelerate release schedules for all five orgs. In each sprint, the team typically addresses 5-10 stories to build out the in-sprint automation work.
In combination with an agile approach, Provar Automation has contributed to a seamless software development lifecycle for the company’s highly customized Salesforce instances, delivering releases with more confidence and fewer defects.
“What are the most helpful features of Provar Automation? Everything has been useful. And, I would say bringing in fields from Excel has been easier – for example to validate a picklist. Also, we’ve been able to connect to different SQL servers and other web services. We’ve also been using the OAuth connection to handle some of our security requirements.”
– Quality Assurance Analyst
The Impressive Results
As a result of switching to Provar Automation, the company is now positioned to accelerate the delivery of new Salesforce customizations, improve end-user and customer satisfaction, reduce risk and consistently improve the quality of each release.
Today, the company is able to test 90% of their use cases and continues to expand the scope of code coverage release after release.
The QA team has also been able to simplify the complexity of testing for their organization using reusable components of tests, easily modifying variables to test in multiple environments and modeling the varying permissions granted to their 2,000+ users.
Another benefit is that the team has been able to save a considerable amount of time – enabling the team to implement in-sprint automation and expand their existing library of regression tests. When the company relied solely on manual testing, it would take one month to test approximately 400 use cases and they can now run 4,000 tests quickly.
“Provar Automation is one of the most sound investments we’ve made to protect our Salesforce infrastructure. And, in turn, it’s really an investment in our customers. Our culture is built around providing world-class customer service and we need our systems to work unfailingly 24/7 to do just that.”
– Quality Assurance Lead
To learn more about how Provar’s solutions can meet your team at all stages of the quality journey, or to share your company’s success story, connect with us today.
