If you’re stuck in the test automation crossroads and are in the process of deciding between which test automation solution to go with to support your Salesforce strategy, we empathize with you. It’s not an easy process — there are many tools on the market today, and all of them boast unique capabilities. We know that there’s no one-size-fits-all approach to testing, and we’re here to help simplify the evaluation process for you.

If you haven’t already downloaded our Product and Vendor Evaluation Guide for Automated Testing, that will be a great resource for your selection process, but we wanted to supplement that guide with some information on robotic versus metadata-driven test building capabilities. These are the two primary approaches test automation tools take today.

In this blog, we’re diving headfirst into a comparison between Provar Automation and tools employing robotic capabilities. We want to ensure that you have all of the right information on hand to make the best decision possible for your team.

Testing Problems and Provar Solutions

Let’s start with the reason why you probably embarked on your evaluation journey — the problems you’re likely encountering. Luckily, we won’t just present the problems. We’ve got the solutions, too!

Here are some common problems you might be experiencing if you’re not happy with your current testing tool (particularly if it employs robotic capabilities) and some ways Provar Automation can serve as a solution.

Problem:

Your test script maintenance is high for your enterprise-level project because common resources can only be accessed from the individual suite level and cannot be communicated with other suites.

Solution:

Use Provar Automation instead, as test cases are modular and reusable so changes only need to be performed in a minimal number of places.

Problem

Your current strategy lacks versatility in integration testing, as your tool limits scenarios to Salesforce, SAP, and ServiceNow, excluding peripheral systems.

Solution:

Use Provar Automation, which provides extensive integration capabilities, allowing connections across various profiles, applications, databases, and Salesforce instances, facilitating end-to-end testing across multiple systems seamlessly.

Problem:

Implementing diverse test scenarios, such as those across countries or roles, demands significant effort within your current testing strategy.

Solution:

Use Provar Automation to help simplify test scenario configuration because it offers standard support for role-based and profile testing. Its dynamic nature enables effortless execution across diverse environments, browsers, languages, and devices with minimal configuration.

Problem:

Your current test automation tool’s low-code, keyword-driven approach faces challenges when handling complex logic or test cases, requiring additional Python coding and libraries.

Solution:

Use Provar Automation, which accommodates various skill levels and preferences by enabling test case construction through intuitive no-code tools like Test Builder or via code (e.g., Page Objects), providing flexibility without compromising on complexity.

Problem:

Accessing resources and educational support for your current test automation tool can be cumbersome, often requiring vendor assistance for problem-solving.

Solution:

Use Provar Automation, as Provar offers comprehensive support with a 24/5 assistance model, an active community forum for networking and learning, free courses and certifications through the University of Provar, Salesforce Trailhead integration, weekly customer standups with a dedicated Success Manager, and potential access to a dedicated Provar Center of Excellence representative for expedited technical issue resolution.

Comparing Provar Automation and Tools Using Robotic Capabilities Side-By-Side

The chart below is a real comparison created by a company in the evaluation process between a test automation tool using robotic testing capabilities versus Provar Automation, which uses metadata-driven test building capabilities. This customer eventually chose to work with Provar due to the functionalities below. This side-by-side is comparable when comparing Provar Automation with other tools using robotic testing on the market, as many of these tools work in a similar fashion.

CapabilityTest Automation Tool Using Robotic Testing (RPA) CapabilitiesProvar Automation
Setup and maintenanceThe tool is maintained by the vendor as it is a cloud-based service. There is no additional maintainability required in terms of tool setup connection and configuration.Test authoring and debugging is performed using a desktop application available for Windows and Mac OS. Test execution can be done on-premise (e.g., local server running Jenkins), in the customers’ private cloud (e.g., AWS EC2 running Docker), or in Provar’s managed cloud environment (Provar Grid). Once set up, maintenance and feature releases are aligned to Salesforce releases, with occasional (optional) releases in between.
Test script maintenanceMaintainability of automated test scripts is high as common resources can only be accessed from the individual suite level and cannot be communicated with other suites. This is a major drawback for enterprise level projects.Test cases are modular and reusable so changes only need to be performed in a minimal number of places. The same test case can be run as different profiles, and in multiple environments and languages. New releases include updates that prevent test cases from breaking due to changes introduced by Salesforce, reducing maintenance work. Provar uses Salesforce’s metadata, which enables test cases to adapt to changes during execution so they are more resilient.
Production functionality testingVery useful for UI testing (good penetration level) with efficient interactive and live UI testing.Setup and teardown test cases allow testers to create test data and clean up afterward, leaving orgs as they were before testing. Test cases can also automatically clean up any records created during execution. End-to-end test cases can span across multiple systems to test complex user journeys.
Data selectionCan be achieved by using basic SOQL or Excel-based manual load.Provar can interact with and utilize data from database queries, SOQL queries (pre-built using declarative interface or custom SOQL), Excel/spreadsheets, CSV, JSON, and any data being pulled from an external REST or SOAP-based API.
Data refreshNot supportedProvar does not natively support data refresh for sandboxes, but you can  utilize Provar’s Apex Bulk feature to populate data in any org as part of a setup or callable test case and either schedule it to run post-refresh, or trigger it manually.
Data integrity testingNot supported (the tool’s focus is to automate UI level testing)DIT can be performed using Provar’s Extract/Assert page layout and SOQL support, which contains necessary field information.
Linking of data to testsNot supportedTests can be made data-driven very simply by utilizing a data read in the test to read from CSV, Excel, JSON, SOQL, Database queries, or REST/SOAP APIs. Provar keeps track of all variables and data used in the testing scope as accessible variables that can be used however you want in your test.
Integration capabilitiesOffers end-to-end testing, but only supports Salesforce, SAP, and ServiceNow (API). It is not possible to test integration and E2E with other peripheral systems.Provar leverages connections that can be built for any profile, application, email database, or Salesforce connection. In an end-to-end scenario, you can have as many connections as you would like in a single test case. This makes the reusability of test cases very easy across multiple profiles and makes it extremely simple to have test cases with multiple applications, which includes having multiple windows open at the same time.
Test evidence formattingAutomatically creates logs and reports for each RUN. These are very detailed and can have screenshots in attachment. Additionally, the tool will capture screenshots by default for failed test cases, and there are video options available.Offers a PDF with screenshots that can be sent via email, JUnit XML, or log file. Integrates with Provar Manager for better analytics and other test management tools.
Configuration of test scenariosThese can be implemented (for instance, across countries or roles), but it takes a lot of effort.Role-based testing or profile testing is standard in Provar. Tests are made to be dynamic by nature and can be run across multiple environments, browsers, languages, and devices with little to no configuration or changes.
Scheduling optionsThis tool does have a scheduling option that can also be triggered from its Devops pipeline.Possible via Provar Manager or any of the CI/CD solutions available on the market.
Monitoring optionsThe tool provides a quality intelligence dashboard that can be customized based on business needs.Provar Manager provides insightful reports and dashboards that can be fully customized.
Script reusabilityFor other domains or regions, scripts are not usable outside of individual suites. If all test cases for a particular suite are a fit for other domains/regions, then they can be reused with manual export and import.Provar has override connections that can easily point a script to another instance. Provar can leverage data-driven testing with variables that make it easy to maintain a single test script for a multitude of variations that are needed to test against (including other languages).
Ease of useThe tool supports a low-code, keyword-driven approach, but it is challenging to write complex logic/test cases as custom python codes and library is needed.Test cases can be built using interactive no-code tools like Test Builder or using code (e.g., Page Objects) depending on the tester’s skill level and personal preference.
Availability of resources/continuing educationFinding the solution to a problem is very difficult and requires the vendor’s support in most cases.Provar offers 24/5 support. The community forum is  available to customers to network and learn more about Provar topics, and the University of Provar provides free courses and certification programs. Provar also offers testing courses on Salesforce Trailhead. Provar provides a Customer Success manager for weekly standups with the customer. In some cases, there is also the option for a separate dedicated Provar Center of Excellence representative for faster resolution on technical queries as well.
ComplexityUI testing when using Salesforce has an advantage as it has its own keyword library dedicated for Salesforce, but the customer is not able to check, and it isn’t clear how it supports other UIs such as AngularJS, React, iframe, etc. Complexity is moderate.Provar utilizes metadata to handle standard Salesforce object locators. Utilizing Provar’s locator technologies, Test Builder is able to penetrate LWC, shadow DOM, iframes, etc. with a simple right click and add to test case. Provar is a full E2E test automation solution and can interact with anything that renders on a browser, APIs, emails, and databases.
Dependency on team resourcesResources are very limited as the tool depends heavily on its support team (see Availability of resources/continuing education).Dependency is low as Provar has online resource support, and most customers maintain their own scripting once the regression has been built.
Dependency on vendorHigh, as a dedicated Python knowledge tester is required for script development and maintainability.Low, as it is all declarative with online support. Provar trains and supports test automation users to be more independent, since they pride themselves on their solution being intuitive and for their users to be proficient quickly.
Cost of ownershipThe tool provides a bundle plan that provides parallel robots, on-premise/desktop robots, mobile testing using browser stack, and a quality intelligence dashboard.Provar’s pricing typically includes developer and execution licenses and  includes 24/5 support. There is no additional cost for API testing.
SaaS/PaaSThis tool is a DevOps and testing solution for low-code SaaS platforms.Provar’s technology stack can be run standalone or integrated with all leading CI/CD, IaaS, PaaS, and Salesforce ISV applications.
Grid functionalityA browser stack license is required if the customer needs to run a device-based automation script using the tool, once test cases are created locally.Provar offers Provar Grid, a fully managed scalable cloud platform for parallel test executions. In addition, Provar Automation provides built-in integrations with Selenium Grid, SauceLabs, BrowserStack, and LambdaTest for remote test execution.

The Main Takeaway

We hope the information above will arm you with the technical insight you need when evaluating test automation solutions. At Provar, we always aim to be transparent about our capabilities and how we work to support you, so having these resources readily available (on our blog, through our white papers, in free University of Provar courses, etc.) is a top priority. But if you do have lingering questions, please don’t hesitate to reach out. We’d love to talk to you in terms of your unique situation!

Want to learn more about how Provar Automation can meet all of your Salesforce testing needs? Request a demo today, and be sure to download our white paper, Robotic Testing: Are You Putting Your Salesforce Environment at Risk?, for an in-depth comparison of robotic versus metadata-driven test building tools!