How Many Pairwise Tests Required (In the Face of Combinatorial Explosions)

This webcast continuing from where, Combinatorial Explosions Explained (Software Testing Example), left off.

This video shows how well designed test plans can achieve full pairwise testing coverage with a very small number of tests. And as the total number of permutations increase exponentially as more parameters and parameter values are added properly designed test plans can provide full pairwise coverage with very few tests.

This is not possible for a person to manually do but using the right algorithms to create a test plans can provide amazing benefits.

Related: More Coverage, Fewer Tests™84% of Software Defects Found in Production Could Have Been Found Using Pairwise TestingLooking at the Empirical Evidence for Using Pairwise and Combinatorial Software TestingHow to Pack More Coverage Into Fewer Software TestsHow Do You Know You are Executing the Right software Tests?

Combinatorial Explosions Explained (Software Testing Example)

It is impractical in many situations for testers to test every possible permutation of software tests.

In this video Justin Hunter, the founder of Hexawise, explains how combinatorial explosions impact software testing. He explains how to quickly calculate the total possible number of permutations exist in simplified systems under test using several examples.

The video ends with a reference to a continuation of this discussion in the next video: How Many Pairwise Tests Required (In the Face of Combinatorial Explosions).

Related: Combinatorial Testing, The Quadrant of Massive Efficiency GainsMind Maps: What, Why and HowCombinatorial Testing for Software

Using Hexawise to Reduce Test Cases by 95% While Increasing Test Coverage

Kathleen Poulsen spoke on using Combinatorial Software Testing at Fidelity Investments during the StarEAST conference in 2017.

Hexawise sponsored an evening gathering with short talks on software testing by various experts. Kathleen discussed the use of Hexawise at Fidelity Investments, as seen in this video.

We hope you enjoy this video. We plan on added several more from the speakers at the evening sessions.

Kathleen, in the video:

We had more than 20,000 less than 30,000 test cases that were in place. 95% of them were redundant; either with the integration layer below them or themselves because people would come on board for 6 months and they would quit and go somewhere else and someone else would come along and reinvent exactly the same test.

I could see that now. When we went with in Hexawise we could see how many tests it actually took and just so small compared to what we had been testing. The first time I ran the full suite of [new Hexawise designed] tests I went back an hour later and I told the manager “we’re done you can move ahead you’re all right now.” By “move ahead” I meant promote to a new environment. And he said, “what, I thought I would see you again in 2 weeks.

That’s the kind of difference it is making.

Related: Examples of the Benefits of Using Hexawise at a Large Bank QA DepartmentHow Do You Know You are Executing the Right software Tests?Too Many Tests

Examples of the Benefits of Using Hexawise at a Large Bank QA Department

Carrie Puterbaugh’s presentation at the Twin Cities Quality Assurance Association on using Hexawise to improve software testing at her organization (a large bank).

In one example she discusses in the video Carrie’s team used Hexawise to create an optimized test suite and provided that to the software vendor to have them run it prior to delivering the software to her bank. In this example historically they vendor was finding 67% of the defects and Carrie’s bank was finding 33%. Now that the vendor is using the Hexawise test suite the vendor is finding 98.5% of the defects and fixing them prior to delivering the software.

Based on these results her team was able to move staff off of testing this application and onto other testing needs of the organization. They are saving 90% of what they used to spend on QA on this project.

Another project she talked about was a high priority and high risk release that they used Hexawise on and achieved the highest quality software release they have ever had.

It was great…
We were able to go to management and say “we reduced the amount of test cases we ran and we got a better quality application.

Related: Combinatorial Software Testing at Fidelity Using HexawiseHow to Think about Test Inputs in Software Testing

How Do You Know You are Executing the Right software Tests?

How Do You Know You are Executing the Right Tests? This video shows a few tools in Hexawise that help answer this question and share the reasoning behind prioritization decisions in software testing (coverage chart, coverage matrix).

Related: How do I understand the testing coverage achieved by my Hexawise-generated tests?More Coverage, Fewer Tests™

2 Minute Introduction to Hexawise Software Testing Solution

This short video gives a quick overview of the Hexawise software as a service solution and a very high level overview of software testing with a focus on interactions between parameters/factors.

Learn more about pairwise and combinatorial software testing practices: Introducing the Hexawise Coverage Matrix!How to Pack More Coverage Into Fewer Software TestsCreate a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas

Too Much to Test and Not Enough Time to Test It All

beginner videoJustin Hunter’s presentation at the 12th annual international software testing conference in Bangalore, India, December 2012.

The techniques discussed focus on how to reduce the amount of time spent selecting and documenting test scripts, reduce the amount of tests needed for execution by creating unusually powerful tests and thus increase the thoroughness of software test suites.

The talk explored the significant risks, for users, companies and employees of failing to catch software application failures before release (for example, looking at releasing Apple Maps with significant failures). And discussed how combinatorial (also orthogonal array or pairwise) software testing can be used to create test plans that test a large number of parameters/factors quickly.

The slides from the presentation:

Continue reading

Mind Maps – What, Why and How

beginner videoIn this video Justin Hunter shows how to use mind maps to clearly and concisely organize and communicate information about your software tests.

You can quickly see mind maps in action by logging into your Hexawise account (it is simple to setup a free demo account, if you don’t have one yet) and open one of the sample test plans. You can make changes to the test plan and export a new mind map and see how the changes are reflected in the mind map.

The video is using Hexwise v 2.0 which is slightly different than the current version on the public site (v 1.x doesn’t have the option to email yourself the mindmap). The main Hexawise site will soon be using Hexawise v2, which includes many enhancements.

As mentioned in the video help.hexawise.com provides guidance and tips on using specific Hexawise features to create software test plans and documentation. As shown in the video to view the mind map you click on the Export option which is in the upper right of the screen. The export option provides several export options including the option to generate mind maps.

Hexawise also can import mind maps that you have already created.

Related: How to Think about Test Inputs in Software TestingUsing Mind Maps for Test Planning