We sympathize with you if you find yourself at a crossroads regarding test automation and are attempting to determine which test automation solution best complements your Salesforce strategy. It’s not an easy process — many tools are on the market today, and all boast unique capabilities. We know there’s no one-size-fits-all approach to testing, and we’re here to help simplify the evaluation process for you.

If you haven’t already downloaded our Product and Vendor Evaluation Guide for Automated Testing, it will be a great resource for your selection process. However, we wanted to supplement that guide with some information on robotic versus metadata-driven test-building capabilities. These are the two primary approaches test automation tools take today.

In this blog, we’re comparing Provar Automation and tools employing robotic capabilities. We want to ensure that you have all the right information to make the best decision possible for your team.

Testing Problems and Provar Solutions

Let’s start with why you probably embarked on your evaluation journey — the problems you likely encounter. Luckily, we won’t just present the problems. We’ve got the solutions, too!

Here are some common problems you might be experiencing if you’re not happy with your current testing tool (particularly if it employs robotic capabilities) and some ways Provar Automation can help.

Problem:


Keeping up with test scripts for your enterprise-level project is important because individual suites cannot access shared resources or trade between suites.

Solution:

Do not use Provar Automation. Because test cases are flexible and can be used again and again, changes only need to be made in a few places.

Problem

Your current strategy lacks versatility in integration testing, as your tool limits scenarios to Salesforce, SAP, and ServiceNow, excluding peripheral systems.

Solution:

Use Provar Automation, which provides extensive integration capabilities. It allows connections across various profiles, applications, databases, and Salesforce instances, facilitating end-to-end testing across multiple systems seamlessly.

Problem:

Implementing diverse test scenarios, such as those across countries or roles, demands significant effort within your current testing strategy.

Solution:

Use Provar Automation to help simplify test scenario configuration because it offers standard role-based and profile testing support. Its dynamic nature enables effortless execution across diverse environments, browsers, languages, and devices with minimal configuration.

Problem:

Your current test automation tool’s low-code, keyword-driven approach can be challenging when handling complex logic or test cases, requiring additional Python coding and libraries.

Solution:

Use Provar Automation, which accommodates various skill levels and preferences by enabling test case construction through intuitive no-code tools like Test Builder or via code (e.g., Page Objects). This provides flexibility without compromising complexity.

Problem:

Accessing resources and educational support for your current test automation tool can be cumbersome, often requiring vendor assistance for problem-solving.

Solution:

Use Provar Automation, as Provar offers comprehensive support with a 24/5 assistance model, an active community forum for networking and learning, free courses and certifications through the University of Provar, Salesforce Trailhead integration, weekly customer standups with a dedicated Success Manager, and potential access to a dedicated Provar Center of Excellence representative for expedited technical issue resolution.

Comparing Provar Automation and Tools Using Robotic Capabilities Side-By-Side

The chart below is a real comparison created by a company in the evaluation process between a test automation tool using robotic testing capabilities versus Provar Automation, which uses metadata-driven test-building capabilities. This customer eventually chose to work with Provar due to the functionalities below. This side-by-side is comparable when comparing Provar Automation with other tools using robotic testing on the market, as many of these tools work similarly.

CapabilityTest Automation Tool Using Robotic Testing (RPA) CapabilitiesProvar Automation
Setup and maintenanceTool is maintained by vendor as it is a cloud-based service so there is no additional maintainability required in terms of tool setup connection and configurationTest authoring and debugging is performed using a desktop application available for Windows and Mac OS. Test execution can be done on-premise (e.g., local server running Jenkins), in the customers’ private cloud (e.g., AWS EC2 running Docker), or in Provar’s managed cloud environment (Provar Grid). Once set up, maintenance and feature releases are aligned to Salesforce releases, with occasional (optional) releases in between.
Test script maintenanceThe maintainability of automated test scripts is high as common resources can only be accessed from the individual suite level and cannot be communicated with other suites. This is a major drawback for enterprise-level projects.Test cases are modular and reusable, so changes only need to be performed in a minimal number of places. The same test case can be run as different profiles and in multiple environments and languages. New releases include updates that prevent test cases from breaking due to changes introduced by Salesforce, reducing maintenance work. Provar uses Salesforce’s metadata, which enables test cases to adapt to changes during execution so they are more resilient.
Production functionality testingVery useful for UI testing (good penetration level) with efficient interactive and live UI testing.Setup and teardown test cases allow testers to create test data and clean up afterward, leaving orgs as they were before testing. Test cases can also automatically clean up any records created during execution. End-to-end test cases can span across multiple systems to test complex user journeys.
Data selectionCan be achieved by using basic SOQL or Excel based manual load.Provar can interact with and utilize data from database queries, SOQL queries (pre-built using declarative interface or custom SOQL), Excel/spreadsheets, CSV, JSON, and any data being pulled from an external REST or SOAP-based API.
Data refreshNot supportedProvar does not natively support data refresh for Sandboxes. You can utilize Provar’s Apex Bulk feature to populate data in any org as part of a setup or callable test case, and schedule it to run post-refresh, or trigger it manually.
Data integrity testingDoes not support; tool’s focus is to automate UI level testing.DIT can be performed using Provar’s Extract/Assert page layout and SOQL support, which contains necessary field information.
Linking of data to testsNot supportedYes. Tests can be made data-driven very simply by utilizing a data read in the test to read from CSV, Excel, JSON, SOQL, Database queries, or REST/SOAP APIs. Provar keeps track of all variables and data used in the testing scope as accessible variables that can be used however you want in your test.
Integration capabilitiesE2E testing; it only supports Salesforce, SAP, and Service Now API; hence it is not possible to test integration and E2E with other peripheral systems.Provar leverages connections that can be built for any profile, application, email database, or Salesforce connection. In an end-to-end scenario, you can have as many connections as you would like in a single test case. This makes test cases reusable across multiple profiles and extremely simple to have test cases with multiple applications, including multiple windows open at the same time.
Test evidence formattingAutomatically create logs and reports for each RUN, which is very detailed and can have screenshots in attachment; also will capture screenshots by default for failed test cases and has video options availableOffers a PDF with screenshots that can be sent via email, JUnit XML, or log file. Integrates with Provar Manager for better analytics and other test management tools.
Configuration of test scenariosThese can be implemented (for instance, across countries or roles), but it takes a lot of effort.Role-based testing or profile testing is standard in Provar. Tests are made to be dynamic by nature and can be run across multiple environments, browsers, languages, and devices with little to no configuration or changes.
Scheduling optionsHas scheduling optionsHas scheduling options via Provar Manager or any of the CI/CD solutions out there, including Copado Devops, Salesforce DevOps Centre, Azure DevOps, etc.
Monitoring optionsThe tool provides a quality intelligence dashboard that can be customized based on business needs.Provar Manager provides insightful reports and dashboards that can be fully customized.
Script reusabilityScript cannot be usable outside of individual suites. If all test cases for a particular suite can be fit for other domains/region, then can be re-used with manual export and import.Provar has override connections that can easily point a script to another instance. Provar can leverage data driven testing with variables that make it easy to maintain one single test script for a multitude of variations that are needed to test against (including other languages).
Ease of useSupport low-code keyword driven approach but to write complex logic/test cases will be challenging; custom Python code and library is needed for that.Test cases can be built using interactive no-code tools like Test Builder or using code (e.g. Page Objects) depending on the tester’s skill level and personal preference.
Availability of resources/continuing educationFinding the solution to a problem is very difficult and requires the vendor’s support in most cases.24×5 support. Community forum is available to customers to network, learn more about Provar topics and Provar University provides lots of courses and certification programs for free. Also Provar testing courses are available on Salesforce Trailhead.
Provar offers a Customer Success manager that is currently doing Provar weekly standups with the Salesforce Automation teams; also there is a separate dedicated CoE for faster resolution to technical queries.
ComplexityUI testing for using Salesforce has an advantage as it has own keyword library dedicated for Salesforce, but we couldn’t check and isn’t about how it supports other UI such as Angular js, React, iFrame, etc. Complexity is Moderate.Provar utilizes metadata to handle standard Salesforce objects locators. Utilizing ProvarX and NitroX locator technologies, Test Builder is able to penetrate LWC, shadow DOM, iFrames, etc. with simple right click and add to test case. Provar is a full E2E test automation solution and can interact with anything that renders on a browser, APIs, emails and databases.
Dependency on team resourcesResources are very limited as tool is new and requires a high dependency on support team.Dependency is low as Provar has online resource support and most customers maintain their own scripting once the regression has been built.
Dependency on vendorHigh, as a dedicated Python knowledge tester is required for script development and maintainability.Low, all declarative with online support. Provar trains and supports test automation users to be more independent, since they pride themselves on their solution being intuitive and for their users to be proficient quickly.
Cost of OwnershipProvides bundle plan which includes parallel robots, on-premise/desktop robots, mobile testing using browser stacks, and a Quality Intelligence Dashboard.Provar’s pricing for is typically for 2 development and 1 execution licenses. Licenses are a floating license model and include 24×5 support. No additional cost for API testing.
SaaS/PaaSThis tool is a DevOps and testing solution for low-code SaaS platforms.Provar’s technology stack can be run standalone or integrated with all leading CI/CD, IaaS, PaaS, and Salesforce ISV applications.
Grid FunctionalityBrowser stack license is required if we need to run our device-based automation script once test cases are created locally.Provar offers Provar Grid, a fully managed scalable cloud platform for parallel test executions. In addition, Provar Automation provides built-in integrations with Selenium Grid, SauceLabs, BrowserStack, and LambdaTest for remote test execution.

The Main Takeaway

We hope the information above will give you the technical insight you need to evaluate test automation solutions. At Provar, we always aim to be transparent about our capabilities and how we work to support you, so having these resources readily available (on our blog, through our white papers, in free University of Provar courses, etc.) is a top priority. But if you have questions, please don’t hesitate to reach out. We’d love to talk to you about your unique situation!

Want to learn more about how Provar Automation can meet your Salesforce testing needs? Request a demo today, and download our white paper, Robotic Testing: Are You Putting Your Salesforce Environment at Risk?, for an in-depth comparison of robotic versus metadata-driven test building tools!