This is the second article in a four-part series, “Why Selenium Users Pick Provar for Salesforce Test Automation.” Don’t just take our word for it. Read along as we look at (1) the behind-the-scenes reality of using Salesforce and why testing is needed in the first place, (2) a comparison of the types of testing available, (3) why Salesforce and Provar are BFFs, and (4) what makes Provar so unique. Read Part 1: One Size Does Not Fit All When It Comes to Test Automation on our blog.
Salesforce is designed around a declarative coding paradigm – “clicks, not code” – meant to enable development by people without traditional coding skills. It’s a critical business application that is updated at least thrice a year, and testing becomes a challenge because the testing tools are often more code-centric than Salesforce itself. There are plenty of ways to meet the testing challenge. Here’s a list of some of the most popular approaches and a summary of their pros and cons.
1. Manual Testing
Pro:
Easy and fast to implement for functional testing. By definition, manual testing doesn’t require coding expertise but business process knowledge and a well-documented test plan.
Cons:
- Error-prone for repetitive testing (large data sets, test permutations).
- Doesn’t scale – the scope of a full regression test for a reasonably-sized Salesforce organization requires a LOT of manual tester hours to complete. One real-world example: 130 hours and 3 weeks of manual regression testing got reduced to 24 hours and 1 day with test automation software.
- Test consistency depends on the tester(s).
- Zero integration with development, CI/CD ecosystem.
2. Coding and Scripting
Pro:
Selenium is the tool to use: It’s open source and has a large community.
Cons:
- Requires technical expertise that takes time to develop. And that expertise is getting more expensive and more complicated to find.
- The test is built separately from the application, requiring an iterative process to code test, run the test, debug a test, and repeat until the test completes.
- The test code needs to be updated whenever the application changes.
- Debugging, updating, and maintaining tests require coding expertise.
3. Record and Playback
Pro:
This is a straightforward method of developing a test; turn on the recorder and step through the processes you want to test – no technical expertise is needed to record.
Cons:
- Tests built with record and playback are easy to break (any changes on the pages or steps), challenging to edit (record a new version), and you need to know the entire process in advance.
- This approach doesn’t scale when building the permutations of the happy path.
4. Scan and Assemble
Pro:
Less removed from the application.
Cons:
- This approach requires tool-specific knowledge to be successful.
- Although it is easier to create tests from elements, more step-by-step configuration is required. It’s like coding (#2 above): There is an iterative cycle of build-run-debug-repeat.
- The test builder has more choices than she needs to create a test step. This means it will take longer to build the test and will not always work correctly the first time.
Additional Considerations
Developer-User Communication Gap
It’s been true forever. An impedance/perspective/vocabulary mismatch exists in conversations between developers and users. For test automation, this means that lots of communication, and typically several iterations of test design, must occur before the test meets the business user requirements.
Test this theorem for size: The more a test-building solution is abstracted from how the application is used (and the application user), the more likely it is to require multiple iterations to resolve a test plan. Manual testing – is less of an issue. The test builder and the application user can be the same person, and it’s easy for a user to see what is being tested. Coding and scripting – big issue. The user can’t read the code to understand or validate the test plan until after the test is debugged enough to run successfully. Iterations are time-consuming and insignificant increments.
Here’s an interesting question to ask when looking at test automation solutions:
Is the test builder more focused on the test building tool, application, and business process?
You must work harder to close the gap if it’s the former.
Building a Complete Test Before Debugging is Problematic
One of the challenges with most traditional web application test-building approaches is that you must build the entire test before attempting to run it. This creates the build-run-debug-repeat cycle noted above. It’s time-consuming to constantly step out of the building process to debug a step and then start over.
Enhancing Generic Web Testing Tech is a Bandaid, Not a Fix
We’ve noted that Salesforce presents some specific challenges to typical web element locator strategies – basically, lots of change and difficult-to-identify fields. There are many technologies in play to help improve the success rate of locators, plus some others designed to reduce the level of abstraction created by the tool. These include natural language, machine learning, OCR, and image recognition. (It’s a blog for another day, but “AI” is often added as a test automation tool feature – whether or not the marketers adhere to a strict definition of AI. Some are spot on, while others are, at best, wildly optimistic.)
Here’s the thing: We’re talking about generic web testing applications, and this approach delivers some innovative, best-in-class solutions. But, for Salesforce, this approach will always fall short because the generic tools are a) creating tests that fail frequently and need a lot of maintenance to keep up with change and b) more abstracted from Salesforce and the Salesforce user. OCR, image recognition, ML, NL, and self-healing tests are all just another way of saying, “We need better web locator strategies.”
If all of this depressingly sounds like you’re out of viable options, you’re not. There’s another approach.
A Test Automation Solution Engineered for Salesforce
Salesforce implementations are essential and complex enough to warrant a tailor-made test automation solution designed specifically for the Salesforce platform instead of adjusting a generic tool to fit. You don’t use a GP when you need a neurosurgeon, you don’t use a plumber when you’re fixing the cooling system of a nuclear reactor, and you shouldn’t use a generic test automation solution on one of the most essential platforms in your business.
One synonym for generic is “off the rack.” Great utility, more affordable, more available. Just not the best fit. A bespoke suit may cost more and take longer, but you will look fabulous.
→ Stay tuned for our third blog series, “Community and Culture Matters.”
Can’t wait for the next blog post in the series, and are already itching to set up a demo? Let’s talk. Connect with us, tell us about your testing needs, and we’ll get back to you to chat further!