As testing professionals that make a living looking for bugs, it can feel even more daunting when the software problem comes from a mistake that was our fault.
Most of the time it’s easy to dust yourself off and move on while learning from the situation, and yet, sometimes a mistake will haunt you forever. It’s also a blunder that makes you double-check in the future while becoming a better software quality champion.
We asked some of our nearest and dearest in the Provar Community about the testing mistake that still haunts them. In addition, we will share five tips to help make future testing less scary.
Remember User Profiles
Not every new feature will be available for every user. That’s why it’s extra important to make sure you’re checking that end users have access to the features that they are anticipating. The feature you’ve created might work beautifully, but can everyone who should be able to access actually use it?
We created a new product for a lower price with fewer features. During go-live, we pushed the new access control on the prod DB – to realize no customers could access their service. The call center experienced it and was swamped. Luckily, it was a DB fix to update. — Jesper Ottosen, 20+ years testing
Don’t forget to check access and user profiles as an end-to-end test to verify that all your hard work can be seen.
Ask the Difficult Questions
You should never assume. Test the tester. All of these things are good advice. Maybe the most important thing about testing is if you see something weird, ask questions … lots of questions.
The memory that haunts me to this day in regards to testing had to do with interactions with people. We had a report of fraud on a test account in production. I was asked to investigate this further. I ended up digging in deep and found a handful of things we needed to address, but one of the things I found was that we were running automation tests in production at a frequency of every 15 minutes. In that test plan, we had a test case that was transacting real company money in the amount of $2. Normally we reverse any transaction performed in production so real money doesn’t get transacted. BUT, this particular automation test wasn’t performing a reversal. Looking into the history, this has been running for the past 3 years.
Doing some math:
60 minutes / 15 frequencyOfTest = 4 times per hour
4 * 24 = 96 times a day
96 * 365 days in a year = 35,040 times a year
35,040 * 3 years = 105,120 times this test has been executed in the last 3 years
105,120 * $2 = $210,240 that has been wasted of our company’s money just executing a test case
The first thing I did was get our automation team to stop that automation from continuing to trigger every 15 minutes since there wasn’t a need for that. From here, I took this to my QA Director, our Engineering VP, and we talked to our fraud department and some managers in finance. No one knew the correct approach on how to address this and reverse these transactions since they have already posted, and many of these were years ago. I then emailed my Finance Executive, CC’d my Director and Engineering VP, and explained the scenario in depth. I wrote to all of the parties I’ve been in touch with that we were out $200k if we didn’t find a resolution for this. The response I got will forever frustrate me … “Talk to your boss.” — Joe, 4 years testing
You might not get an answer that is helpful, but at least you’ve learned to keep an eye out!
Setting Yourself Up for Success
Not every day is a peak mental acuity day. Maybe you were up late the night before helping your kids with their science project or watching the last few episodes of your favorite TV show. Personal distractions can drain our focus just as easily as a heavy workload.
On one of my last major projects, I often did my testing late at night when the devs had pushed changes and UAT was up-to-date. Anyone who knows me can attest to the fact that my brain has run out of usable brain cells by then, so I’m not sure why I ever thought this was a good idea. This particular sprint’s story was only a net-new addition to existing functionality. Nothing should have changed, so in my haste to complete the work, I gave my tests a run and passed it with flying colors.
The next morning I woke to an email from the BA asking if I had passed the wrong story because there were a ton of bugs he’d found and logged against it. Apparently the new changes broke all of the old functionality, which I had not looked at, and tumbled everything that relied on those previous paths. Oops. — Tara, 10 years testing
Scheduling out tasks so you’re not in a rush is always a recommended practice. Set yourself up for success by making sure you have the resources and time to do the job right.
Use All the Test Steps
I know one of the greatest enemies of testing is complacency, especially when we’re doing the same tasks over and over again. It’s easy to cut corners or to think you did something because you’ve done it a million times.
I forgot to change the emails of contacts in a Sandbox and sent emails to all contacts while doing regression testing. It was all contacts. — Andrew, 7 years in software quality
Make sure you follow your own test steps to the letter. At best, you might be adding in a variable you don’t need, but worst case scenario could be a very public blunder.
Own It
Tests pass for years without so much as a thought. Until they don’t. I once joined a team that just assumed their tests had been passing, but no one was looking at the output to see over 70% failure rates.
The year was 2009 and I had recently joined the first IT firm as a software tester. Working as a [beginner] in the software testing industry, I was assigned testing of a mail utility for a large banking project. Unbeknownst to me, the testing process included a detailed and lengthy setup of the mailbox on a local server. The weather in Bangalore was gloomier than usual and my friends started logging out of the office as darkness of the night descended upon us. I asked a senior tester, “Do you know how to set up the mail server for this test case?”, to which he responded: “Don’t worry at all, that test case is pretty old and always works, so go ahead and mark it passed.” At that moment, I made one of the testing mistakes which still haunts me. I marked the test “passed” without actually running it … the next day we had an unusual meeting in which our manager highlighted that the mail utility test had actually failed in the latest build and the test case was not working. — Robin, 13 years testing
A test that no one checks the outcome of is not a test at all. It’s just a time-wasting activity.
It’s Not So Scary
Testing isn’t always so scary, and even the most seasoned professional can have questionable moments that will haunt them. Take heed and remember there is no mistake that cannot be recovered from. As John Wooden reminds us, “The team that makes the most mistakes usually wins, because doers make mistakes.”
I like to remind people that it’s always important to do things at the right pace to help avoid the worst mistakes, but to err is human. We hope that this resource reminds you to share your mistakes openly (even the scary ones), and help build a culture of continuous and inclusive learning for your teams.
Have a testing mistake that still haunts you? Share this spooky testing resource on social and let us know your story! For more on Provar Community and the various test strategy resources it offers, including its Community forum, visit the Community page.
