How Do You Know You are Executing the Right Tests? This video shows a few tools in Hexawise that help answer this question and share the reasoning behind prioritization decisions in software testing (coverage chart, coverage matrix).
The Hexawise coverage matrix displays pairwise coverage as each test is run in the test plan. The Hexawise created test plans also put the tests in order that provides the most coverage as quickly as possible (so if you have to release before you finish the test plan you have achieved as much coverage as possible with the tests you managed to execute).
Subscribe to the
- RSS feed for the Hexawise.TV blog
- Hexawise YouTube Channel
- RSS feed for main Hexawise software testing blog
Kathleen Poulsen of Fidelity Investments gave a presentation at STAREAST 2017 sharing her experience using Hexawise to improve their software testing performance.
We were able to reduce from over 12,000 test to 600 tests.
Those 12,000 tests provided unknown but less than full pairwise test coverage. Using Hexawise Fidelity generated just 600 test cases that provided complete pairwise testing coverage.
A big gain for Fidelity was that the testing teams using Hexawise were able to compare and coordinate testing coverage. For example, previously the UI and services teams could not effectively communicate test plan coverage and that resulted in a great deal of duplicated testing. Now they can communicate with each other easily and coordinate their testing plans using Hexawise.
This interview with Mike Bland is part of our series of “Testing Smarter with…” interviews. Our goal with these interviews is to highlight insights and experiences as told by many of the software testing field’s leading thinkers.
Mike Bland aims to produce a culture of transparency, autonomy, and collaboration wherever he goes, in which “Instigators” are inspired and encouraged to make creative use of existing systems to drive improvement throughout an organization. The ultimate goal of such efforts is to make the right thing the easy thing. He’s followed this path since 2005, when he helped drive adoption of automated testing throughout Google as part of the Testing Grouplet, the Test Mercenaries, and the Fixit Grouplet.
Hexawise: If you could write a letter and send it back in time to yourself when you were first getting into software testing, what advice would you include in it?
Mike: When I first started practicing automated testing and had a lot of success with it, I couldn’t understand why people on my team wouldn’t adopt it despite its “obvious” benefits. One of the biggest things that experience, reading, and reflection has afforded me is the perspective to realize now that different people adopt change differently, at different rates and for different reasons, and that you’ve got to create the space for everyone to adapt accordingly.
As I say in my most recent presentation, “The Rainbow of Death”, metrics and arguments are far from sufficient to inspire action in either the skeptical or the powerless, and the greater challenge is to create the cultural space necessary for lasting change.
Oh, and you’ve got to repeat yourself and say the same thing different ways multiple times—a lot.
Hexawise: What change management lessons did you learn while driving adoption of test automation methods at Google between 2005-2010? Which of those lessons were applicable when you were involved in the recent U.S. federal government effort to bring in talented tech people to bring new ways of working with technology into the government? Which of those lessons were not?
Mike: The top objective is to make the right thing the easy thing. Once people have the knowledge and power to do the right thing the right way, they won’t require regulation, manipulation, or coercion—doing things any other way will cease to make any sense.
Most of what I learned in terms of specific approaches to supplying the necessary knowledge and power has come from trying different things and seeing what sticks—and I’m still working to make sense of why certain things stuck, years after the fact. The most important insight, as mentioned earlier, is that different people adopt change differently, for different reasons, and as a result of different stimuli. Geoffrey A. Moore’s Crossing the Chasm was the biggest eye-opener in this regard.
Then, years later, when I saw fellow ex-Googler Albert Wong present his “Framework for Helping” to describe his first experience in the U.S. Digital Service, I instantly saw it snapping in place across the chasm, describing how the Innovators and Early Adopters from Moore’s model—who I like to call “Instigators”—need to fulfill an array of functions in order to connect with and empower the Early Majority on the other side of the chasm. Of course, through the filter of my own twisted sense of humor, I thought “Rainbow of Death” might make the model stick in people’s brains a little better.
This short video gives a quick overview of the Hexawise software as a service solution and a very high level overview of software testing with a focus on interactions between parameters/factors.
Learn more about pairwise and combinatorial software testing practices: Introducing the Hexawise Coverage Matrix! – How to Pack More Coverage Into Fewer Software Tests – Create a Risk-based Testing Plan With Extra Coverage on Higher Priority Areas
Justin Hunter’s presentation at the 12th annual international software testing conference in Bangalore, India, December 2012.
The techniques discussed focus on how to reduce the amount of time spent selecting and documenting test scripts, reduce the amount of tests needed for execution by creating unusually powerful tests and thus increase the thoroughness of software test suites.
The talk explored the significant risks, for users, companies and employees of failing to catch software application failures before release (for example, looking at releasing Apple Maps with significant failures). And discussed how combinatorial (also orthogonal array or pairwise) software testing can be used to create test plans that test a large number of parameters/factors quickly.
The slides from the presentation:
In this video Justin Hunter shows how to use mind maps to clearly and concisely organize and communicate information about your software tests.
You can quickly see mind maps in action by logging into your Hexawise account (it is simple to setup a free demo account, if you don’t have one yet) and open one of the sample test plans. You can make changes to the test plan and export a new mind map and see how the changes are reflected in the mind map.
The video is using Hexwise v 2.0 which is slightly different than the current version on the public site (v 1.x doesn’t have the option to email yourself the mindmap). The main Hexawise site will soon be using Hexawise v2, which includes many enhancements.
As mentioned in the video help.hexawise.com provides guidance and tips on using specific Hexawise features to create software test plans and documentation. As shown in the video to view the mind map you click on the Export option which is in the upper right of the screen. The export option provides several export options including the option to generate mind maps.
Hexawise also can import mind maps that you have already created.
This is an 8 part series of videos showing, in detail, various steps that can be used in creating a pairwise software test plan. Many test plans will not require many of these optional steps. Most of the videos have run times of 1-4 minutes.
The series of videos looks at how to create a set of functional test cases for customers purchasing products on Amazon.com using the Hexawise, a tool to create software test plans with pairwise and combinatorial test coverage.
Video 1 of 8, getting started creating up a new test plan.
Video 2 of 8, marking invalid pairs. This allows a test planner to avoid pairs that are not sensible – for example, if a certain product is not available for payment with a check (perhaps a service that requires a monthly payment). Hexawise will then avoid creating any test conditions that contained the invalid pairs that were identified.
This video provides practical tips for selecting appropriate test inputs for pairwise and combinatorial software test design.
Our experience shows as software testers begin to use pairwise test design strategies selecting the correct input parameters to test is often a bit of a struggle. This video will help testers get up to speed with designing pairwise software test plans effectively.
A basic introduction to pairwise software testing, designing a set of pairwise tests for a simplified banking application, and answering the following questions:
What is pairwise testing?
What challenges does pairwise testing solve?
How do pairwise software test design methods prioritize which specific test conditions are included in each of the tests in the plan?
What two aspects of pairwise test solutions don’t receive as much attention as they deserve? (Variability between tests, and front-loading of combinatorial coverage)
Video 2 of 2:
This video demonstrates:
- How should you think about including parameters and values in your inputs so that they trigger specific business rules?
- How can you remove “impossible to test for” scenarios from presenting?