“We have never seen a UI testing tool that has assisted so much and enabled us to make a fully functional test suite within four months from the first time we got access.“
Nicolai Rønnow Iversen
Technical SalesForce Architect, Alm. Brand
How Alm. Brand Leverages Provar at a Glance
150 Provar test cases running nightly within two test orgs. Currently testing Salesforce Sales Cloud and Service Cloud, including Console, Lightning testing, Flows, Visualforce, and many integrations.
The Challenge of Using Provar with GitLab and Docker
To support a fully-functional agile software development methodology, the Alm. Brand needed to leverage test automation capabilities with GitLab and Docker to deliver a new release on a 14-day cycle and keep improving multiple times a day.
As part of Alm. Brand’s dedication to delivering the best experience possible for users, quite an extensive automation suite was built up, acting as a regression test suite when promoting code from one test environment to another. When Salesforce significantly updated the user interface, transitioning from classic to Lightning, this suite was no longer compatible. The Alm. Brand team was forced to use manual testing processes, which required 120+ hours per regression cycle.
The Solution to Manual Testing
After thoroughly evaluating other automated software testing applications, the Alm. Brand worked with House of Test, the confessed Rebels of IT, which provide expert test consultancy for clients across Sweden, Denmark, Switzerland, and the U.K. to select and implement Provar to simplify and accelerate the process of software innovation.
Test Case Implementation
Automated tests that rely on UI are naturally slower to execute than tests that rely only on APIs. As Provar supports both UI access and makes SOQL/API integration easy, all tests in Alm. Brand’s automation suite uses a minimum of UI components.
The tests will typically set up the test data using API access, then run a short user scenario with UI components. Finally, verify the results with API access to support the strategy based on the testing triangle. Alm. Brand can use the test suite across environments and the same test with multiple users to verify the difference.
Continuous Integration and Containerization
Even with these optimizations, the regression suite takes a considerable time to run. To cut down on this time, the Alm. In collaboration with the House of Test, Alm. Brand configured the GitLab pipeline to run several tests in parallel using multiple Docker containers. The parallelization is limited by how many Docker containers GitLab can handle in parallel.
Alm. Brand designed the test cases to be self-contained to allow running all test cases in parallel. Given this, the overhead to set up the test basis for each test case and remove it is won multiple times by being able to execute it in parallel.
Using Provar’s support for automating tests with Ant, Alm. Brand built a pipeline in GitLab that enables the Alm. Brand teams to work collaboratively on the automation suite and get fast feedback, both when committing new code and promoting releases to new environments.
The pipeline is based on Docker continuous integration containers that contain the Provar tool, Ant, and Chrome. GitLab uses these containers to run small smoke tests when new code is checked in, followed by a more extensive suite when code is merged. Every night, a full regression suite is run, in several environments, so every morning, the teams at Alm. Brand can get feedback if any new deployment has caused any regressions.
For versioning of test cases, the Alm. Brand used Gitlab to design a branching structure to create multiple versions of the same test case so the test case in the environments followed the actual code it was testing.
The Results with GitLab and Docker
As a result, Alm. Brand now has a much more robust regression test suite that can give fast feedback, easily be maintained, and is executed automatically. This not only saves Alm. Brand many hours of manual testing for each release, but also increases the confidence of the stakeholders and development teams in Alm. Brand each time new code is released, improving the quality and the speed of the deliveries.
Stats at a Glance
The Alm. Brand was able to get up and running with Provar in less than four months.
By implementing Provar’s automated testing capabilities, the team can save 120+ hours of manual testing per regression cycle.
“One of the reasons Alm. Brand has been so successful in their implementation that they focused on creating tests to keep maintenance minimal.“
“They optimally created tests to reduce future rework, adopting features such as callable tests, API testing, and data-driven testing into their overall architecture to underpin their UI testing. This enabled a fast and efficient delivery of a test suite that will remain effective into the future.“
Customer Success Manager, Provar
“With Provar, we did not just succeed and create a foundation for our automated testing, but also a foundation for our goals to the CI implementation to support our Agile strategy.
We could support the idea of moving the test down in the test pyramid as much as possible while keeping our test coverage intact. One of the essential things that have made this potential has been how Provar treats its customers, not only helping to solve the problem but also listening and receiving advice from the customer on what can make Provar a better tool.
They are not afraid of being challenged and, at the same time, bringing their ideas to the table. That part has made communication between Alm. Brand and Provar is one of the leading pillars that have made us reach where we are in just four months.“
Technical Tester, Alm. Brand