Test Design Techniques on Real Examples

Emma Khomits

·

Follow

20 min read

·

Mar 31, 2024

--

1

Listen

Share

In the previous article, we considered the difference between test analysis and test design and defined a step-by-step test analysis algorithm. In this article, we will take a detailed look at the fundamental test design techniques.

Some techniques are simplified, and some specific details are skipped for better understanding. If you want to dive deeper, the ISTQB Test Analyst guide is recommended.

Conventionally, test design techniques can be divided into two categories:

• Techniques that help to generate an optimal set of test data to verify the processing of values coming into the system.
• Techniques that help check the correctness of the business logic of the system.

Here are the techniques that we will learn further:

Data Verification

Equivalence Partitioning

Equivalence partitioning divides data into partitions (also called classes) based on the assumption that data in the same partition is processed in the same way by the system and produces the same result.

The theory behind this technique is that if a test case, that tests one value from an equivalence partition, detects a defect, this defect should also be detected by test cases that test any other value from the same partition. Therefore, one test for each partition is sufficient.

Let’s apply this technique to the Create Account feature in the Slack application.

To create an account, the user needs to submit a form that has the following parameters with defined requirements:

• Full Name
  - 1–50 characters
  - Required
• Password
  - 6–30 characters
  - Required
• Email Newsletter
  - Yes — by default

By the requirements, the Full Name parameter value should accept from 1 to 50 characters, we can make the reasonable assumption that any value with characters from 1 to 50 will be handled the same way, meaning that the same code will be exercised.

Since all the values from 1 to 50 characters are in the same equivalence partition, meaning that each test using one of those values will receive the same processing, we need to test only one value in that partition. Instead of 50 test cases, each with a different value, we need only one to test the correct handling of any value in that partition.

This partition is called valid because it contains valid values that the system should handle normally.

There are also invalid partitions, that contain values that the system should reject or at least escalate to the user for correction. Invalid partitions for the Full Name parameter are above 50 characters and below 1 character which means empty for text inputs.

For the Password parameter, we have the following partitions based on the requirements:

Here is a set of test data for all parameters in our registration form:

Full Name/Password:

• One positive value from a valid partition.
• One negative value from each invalid partition.
• Note, that for the Password parameter, an empty value is added additionally, as it always makes sense to check the “0” value separately.

Email Newsletter:

• This parameter doesn’t have invalid partitions, it’s just a Boolean value.

As Full Name and Password parameters have text type input data, except the length of the value, we also should consider the content itself, which can include:

• Letters
• Numbers
• Symbols

In addition, letters can be in different languages, so each alphabet type can be considered as a subclass of the Letters class, e.g., we might decide to check the following alphabets:

• Latin
• Cyrillic
• Asian

Let’s say we have the following additional requirements:

• Full Name parameter: accepts all alphabets and numbers, doesn’t accept symbols.
• Password parameter: accepts only a combination of Latin letters, numbers, and symbols.

Here is how our data set will look now:

Note that this is just a set of values, we will move on to the test cases developing later, after considering the next technique.

Applicability

Equivalence partitioning technique is appropriate when all the members of a set of values to be tested are expected to be handled in the same way.

Partitions can exist in many places and are not limited to ranges. Partitions can also be made for something as simple as Boolean values — Yes/True is in one partition and No/False is in another one, as we have for the Email newsletter parameter.

Also, partitioning can be applied not only to input data but also to output data, test environments, OS and browser types and versions, hardware configuration, etc. Basically, it can be applied to anything where variance can affect the outcome of a test.

Types of Defects

For the most part, we are looking for a situation where some equivalence partition is handled improperly. That could mean the value is accepted when it should have been rejected or vice versa, or that a value is properly accepted or rejected, but handled in a way appropriate to another equivalence class, not the class to which it actually belongs.

Drawbacks

If the assumption is incorrect and the values in the partition are not handled in exactly the same way, this technique may miss defects.

It is also important to select the partitions carefully. For example, an input field that accepts positive and negative numbers might be better tested as two valid partitions, one for the positive numbers and one for the negative numbers, because of the likelihood of different handling. Depending on whether or not zero is allowed, this could become another partition.

The next technique is the Boundary Value Analysis which is used in combination with Equivalence Partitioning.

Boundary Value Analysis

Boundary value analysis technique is used to test the proper handling of values that exist on the boundaries of equivalence partitions. The boundaries are defined by the maximum and minimum values in the defined equivalence partition.

Why are we interested in the boundaries and what is the point of these extra tests? There is a hypothesis that bugs frequently occur in the handling of the edge conditions.

There are two approaches to the this technique:

Two-value boundary

With two-value boundary testing, the boundary value itself and the value that is just outside the boundary are used (the smallest possible increment that is outside the boundary). For example, for amounts in a currency that has two decimal places, if the partition includes the values from 1.00 to 10.00:

• the lower boundary test values are 1.00 and 0.99
• the upper boundary test values are 10.00 and 10.01

Three-value boundary

For three-value boundary testing, the values before, on, and over the boundary are used. For the same example, if the partition includes the values from 1.00 to 10.00:

• the lower boundary test values are 0.99, 1.00 and 1.01
• the upper boundary test values are 9.99, 10.00 and 10.01

Three-value boundary approach provides more thorough coverage and is recommended to be used for the higher-risk functionality, but it also takes more time. The decision regarding whether to use two-value or three-value boundary testing should be based on the risk associated with the item being tested.

Let’s apply a two-boundary value approach to the parameters in the registration form from the example above:

Full Name

• Valid partition: 1–50
  - the lower boundary values: 1 and 0
  - the upper boundary values: 50 and 51
• Invalid partition: 0
  - it has only one value
• Invalid partition: 51 and more
  - the lower boundary values: 50 and 51
  - the upper boundary values: equals to the maximum value that can be entered in the field input. In our case, the field does not allow entering more than 50 characters, so it will be enough to check the values of the lower boundary.

Password

• Valid partition: 6–30
  - the lower boundary values: 6 and 5
  - the upper boundary values: 30 and 31
• Invalid partition: 0–5
  - the lower boundary values: 0 only (there is no value outside)
  - the upper boundary values: 5 and 6
• Invalid partition: 31 and more
  - the lower boundary values: 30 and 31
  - the lower boundary values: similar to the Full Name field, it is enough to check the values of the lower boundary.

You might note that a value outside of one boundary is often a boundary in another equivalence partition. If so, there is no reason to duplicate them in the tests.

Here is our expanded set of data with boundary values:

Applicability

Boundary value analysis is an extension of Equivalence Partitioning that applies only when the members of an equivalence class are ordered. An ordered set is one where we can say that one member is greater than or less than some other member if those two members are not the same. We have to be able to say this meaningfully, too. Just because some item is right above or below some other item on a dropdown menu does not mean that these two items have a greater-than/less-than relationship.

Ordered equivalence partitions are required because of the concept of being on and off the boundary. If the ordering is not relevant from the business or technical point of view, then boundary values should not be in focus.

In the registration form example, we have two partitions for the Email Newsletter parameter — Yes and No, we can’t use this technique for it because there aren’t defined edges of the partitions. You are either in it or not.

In addition to number ranges, partitions for which boundary value analysis can be applied include:

• Numeric attributes of non-numeric variables (e.g., length).
• The number of loop execution cycles.
• The number of iteration elements in stored data structures such as arrays.
• The size of physical objects (e.g., memory).
• The duration of activities.
• Date and time values.

Types of Defects

This technique finds defects regarding the handling of the boundary values, displacement, or omission of boundaries, particularly errors with less-than and greater-than logic.

Drawbacks

Because the accuracy of this technique depends on the accurate identification of the equivalence
partitions in order to correctly identify the boundaries, it has the same difficulties.

Another drawback of doing boundary value analysis is the risk of putting too much emphasis on the edge cases and not enough on the rest of the functionality. We have to balance the time we spend against the risk we can mitigate.

Composing Tests

Once we have a set of test data for each parameter we can derive test cases. Checking each parameter value individually is inefficient and takes more time. We will use combinatorial testing to combine values of different parameters into tests which will allow us to have fewer tests with more coverage.

There is a rule to not combine multiple invalid values in a single test to prevent a situation where the presence of one invalid value might mask the incorrect handling of another invalid value. Or we might get an error that says “invalid value.” but which value was detected as invalid? When designing test cases using invalid partitions, it’s important to be sure that there will be a clear result from the test that will show the proper handling of all the partitions.

So, first, we need to create a set of positive tests by selecting a valid value for each parameter. The number of test cases is equal to the biggest number of test values within a parameter, in our example, it’s Full Name with four values in the content partition, so we get four positive test cases.

Positive tests

Next, we create a set of negative tests where each test will have an invalid value for one parameter and a valid value for every other parameter. The number of test cases is equal to the number of all invalid test values in the data set, which is 10 in our example.

Negative tests

You can notice there are no specific values in our test cases. Test design techniques give us high-level test cases but we still need to provide concrete values and expected results during actual testing.

Such high-level test cases are reusable across multiple test cycles with different concrete data, which is a good approach for regression testing as every time we can use different values that help to avoid pesticide paradox (one of the testing principles that says: tests with the same values are no longer effective at finding defects, just as pesticides are no longer effective at killing insects after a while.)

In the test cases above, values are randomly combined, values are randomly combined but our combinations may not match users’ combinations and we may miss bugs. It might not be the case for small registration forms like in the provided example, but in complex applications, there can be objects with a large number of parameters, each having multiple possible values which gives rise to more combinations.

The only way to know for sure if there is a bug is to test all the possible combinations, but that’s impossible in the time allowed.

For those cases, we can use the Pairwise Testing technique, which allows us to control the combinatorial explosion, and efficiently test a large number of possible combinations of parameters with a relatively small number of test cases.

Pairwise Testing

Pairwise testing is a technique in which test cases are designed to execute all possible combinations of each pair of input parameters. Pairwise testing ensures that each value of one parameter gets tested with each value of another parameter instead, of testing all possible combinations between parameters.

All pairs coverage is a lot easier to achieve than all combinations. For instance, if you want to test 4 parameters each having 5 values, all combinations lead to 625 tests. All pairs testing does it in 32 tests.

There are various tools available online that can help automate the process of designing test cases (see https://www.pairwise.org/tools.html), we just need to prepare test data for the tool, and then the tool will generate a set of test cases that cover all possible combinations of parameter-value pairs.

Let’s apply this technique to the Notifications Preferences feature in Slack, using the Allpairs tool.

Step 1: Prepare a list of parameters with test values

First, we need to prepare a list of the parameters and values we want to test in a table view.

Here are all notification parameters and possible values:

Step 2: Run the tool with prepared test data

Next, we will run the tool and feed our data to it (the tool instruction explains how to do it).

Here is a set of test cases the tool generated:

We got 15 test cases that guarantee that each value of each parameter is paired in at least one test.

Cells that are marked with a “~” symbol mean that we can substitute any other value because all of its pairings have already been achieved. We might substitute those values with more commonly used values or values that are common sources of defects.

Now, the output of the tool can be used as input for test cases but we still need to provide the expected result for each combination.

Applicability

Pairwise testing technique is useful for testing functionality with a large number of input parameters with multiple possible values, but it’s not limited to it.

We can also use the Pairwise testing technique when the behavior of the system is affected by different factors or configurations, and defects may arise due to specific combinations.

For example, for a web application, the following factors might be considered during testing:

• Operating system
• Browser
• Application language
• Authorization method

We can apply the same tool to generate test cases covering all pair combinations.

Here is a list of parameters with values:

Here are the generated test cases:

Reviewing the output, we can see that there are some impossible combinations in TC 2 and TC 6 — Mac OS with Edge and Windows with Safari. So we need to remove them but we must ensure that other parameter combinations in those rows (Language and Authorization) are met in other tests.

We can make the following changes:

• TC 2: to keep the Spanish and Email/Password pair, we can use TC 13 and replace Windows with Mac as Windows has the “~” label which means it’s not required there.
• TC 6: to keep the Spanish and SSO pair, we can use TC 14 — keep Windows & Firefox & Spanish values, and replace Social Network with SSO.

The final table looks like this:

Type of Defects
The most common type of defects found with this test technique are defects related to the combined values of two parameters.
Incorrect handling of combinations and discovery of combinations that interact when they should not or bugs that might occur due to interactions between different input parameters.

Drawbacks
The major limitation is the assumption that the results of a few tests are representative of all tests and that those few tests represent expected usage. If there is an unexpected interaction between certain parameters, it may go undetected with this test technique if that particular combination is not tested.

To minimize risk, it’s important to review the generated combinations and augment them as needed with knowledge of customer preferences, previous failure information, and known common configurations.

Business-logic Verification

State Transition Testing

State transition testing technique is used to test the ability of the test object to enter and exit from defined states via valid transitions, as well as to try to cover invalid transitions.

It allows us to focus on the different states of the object and transitions between them, rather than testing individual features in isolation.

Here’s how state transition testing technique works:

Step 1: Identify States

• States reflect the various stages an object can be in during its lifecycle.
• Each object can have a different state depending on what we do with it.
• The state can change over time due to interactions, processes data or actions performed on the object.
• States should not intersect, an object cannot be in two states at once, and at the same time, it always exists in at least one.
• Example: object User can have states like Invited, Active, Deleted.

Step 2: Identify Actions

• Actions cause the test object to transition from the current state to a new state or from the current state back to the current state again.
• Actions can be user actions, system responses, or any external stimuli.
• Example: actions for User object can be: Invite, Register, Delete.

Step 3: Define Transitions

• A “transition” refers to a change of state that occurs when an action takes place.
• Example: User object might transition from the Invited state to the Active state when the user completes the registration process.

Step 4: Design State Transition tests cases

The State Transition Testing technique can be represented as a diagram or as a table. We will look at both variants.

Let’s apply this technique to the Message object in the Slack application:

States:

• Draft
• Scheduled
• Sent
• Deleted

Actions:

• Send
• Schedule
• Edit
• Delete

Now we are ready to draw a state transition diagram. The common notations include bubbles for states and arrows for actions.

Next, we can derive test cases. The minimum coverage should include verifications of all states and transitions:

• TC 1: Draft — Sent — Deleted
• TC 2: Draft — Scheduled — Sent
• TC 3: Draft — Scheduled — Deleted
• TC 4: Draft — Deleted

As mentioned above, a state-transition diagram is not the only way to document the object’s behavior. The diagram may be easier to comprehend, but the state transition table may be easier to use in a complete and systematic manner.

State transition tables consist of four columns:

1. Current State: a list of all states of the object.
2. Action: a list of all actions for each state.
3. New State: the object state after the action is occurred.
4. Expected Result: how the system should respond to the action.

Here is how a state transition table might look for the Message object:

The advantage of a state-transition table is that it lists all possible state-transition combinations, not just the valid transitions as the diagram does.

Creating state-transition tables often unearths combinations that were not identified, documented, or dealt with in the requirements and this is a good moment to find out what exactly should happen in each of these situations.

Checking valid and invalid transitions should be required for safety-critical software or high-risk systems such as avionics, medical devices, etc. and it’s a good start for security testing as well.

Applicability
State transition testing is applicable for any software that has defined states and has actions that will cause the transitions between those states.

It’s an excellent tool to capture certain system requirements, namely those that describe states and their associated transitions.

Note that the State transition testing technique should be applied to one specific object. It describes the states of a specific object, the actions that affect this object, and the transitions of the object from one state to another. A common mistake is to mix different objects into one state transition diagram/table.

Types of Defects

This technique can find omissions and contradictions in requirements before implementation. Typical defects include the following:

• Inability to reach some states.
• Inability to execute valid transitions correctly.
• Ability to execute invalid transitions.

Drawbacks

The effectiveness of this technique relies on accurately identifying and representing the states of the objects. If the identification of states is incorrect, the test cases derived from the state transition model may not accurately reflect the system’s behavior.

Decision Tables

Decision tables are an excellent tool for capturing complex business rules based on a set of conditions and related actions that will occur for specific combinations of conditions.

By using a decision table, all the possible input conditions and corresponding output actions can be easily represented and analyzed, making it easier to identify any gaps in the testing coverage.

Let’s create a decision table for the Notifications feature in Slack when a message is sent to a channel.

Step 1: List a set of conditions

We have the following conditions that affect notifications:

• Notification settings are on or off.
• Do not disturb mode is on or off.
• The channel is muted or not.

Step 2: List associated actions

There are different types of notifications that can be sent:

• Banner preview.
• The channel name gets bold in the sidebar.
• The application icon gets a red dot.

Now we can start creating a decision table, putting the conditions on top and the associated actions at the bottom.

Step 3: Define the number of columns with decision rules

Next, we can define the number of columns (decision rules). To get the number of columns we need to multiply the number of possible values of each condition. In our example, we have three conditions and each condition has two possible values — yes or no, so it will be 2 x 2 x 2 = 8 columns.

Step 4: Populate columns with decision rules

First, we will populate Conditions rows using the following pattern:

• The first condition: half of the columns are Yes, then half No.
• The next condition: quarter Yes, then quarter No, then quarter Yes, then quarter No.
• The last condition: the alternation is Yes, No, Yes, No, etc.

Then we need to populate Actions that should be taken or not taken depending on the combinations of conditions in each column.

As a result, we got a table where each column is a rule that defines a unique combination of conditions that result in the execution of the actions associated with that rule.

Creating test cases from this decision table is simple — each rule (column) becomes a test case, where the Conditions specify the inputs and the Actions specify the expected results.

In the above example, conditions have Boolean values (Yes/No, False/True). Decision tables in which conditions are strictly Boolean are called limited-entry decision tables.

Sometimes, conditions are not Boolean and can take multiple values, in which case a larger number of combinations will be possible. Such tables are called extended-entry decision tables.

Let’s consider a feature like Transfer Money where the transaction fee depends on the following conditions:

• Card Type: Standard, Business, Premium
• Transaction Amount: less than 1k, between 1k and 10k, more than 10k
• Foreign exchange: Yes or No

And the following actions can be taken:

• Transaction fee
• Conversion fee

Next, let’s count the number of columns for the table. The first condition has three values, the second condition has three ranges of values and the last condition has two values. To get the number of rules we need to multiply them — 3 x 3 x 2 = 18.

Finally, we need to populate associated actions where the amount of the fees depends on the combination of conditions:

As we learned before, we can use each column as a test case, but you may notice that the Transaction Amount condition has a range of values in each column, and during testing, we will need to use specific values. To choose specific test data for each range we can use Equivalence Partitioning and Boundary Value Analysis techniques. As a result, we will get more than one test case per column, for example, for column 1 we can have the following test cases:

• TC 1: Card Type: Standard, Transaction Amount: 500, Foreign exchange: Yes
• TC 2: Card Type: Standard, Transaction Amount: 1, Foreign exchange: Yes
• TC 3: Card Type: Standard, Transaction Amount: 999, Foreign exchange: Yes
• TC 4: Card Type: Standard, Transaction Amount: 0, Foreign exchange: Yes

When trying to test every possible input combination according to the conditions, decision tables can become very large. If there are many conditions, exercising all the decision rules may be time-consuming, since the number of rules grows exponentially with the number of conditions. In such cases, to reduce the number of rules that need to be checked, a minimized decision table or a risk-based approach may be used.

The following recommendations can help to collapse the decision table:

• Deleting columns containing infeasible combinations of conditions.
• Merging columns, in which some conditions do not affect the actions.
• Picking the most frequently used condition first and considering all the other conditions, based on risk analysis.

Applicability

Decision tables technique is a valuable tool for representing business rules and managing complex decision-making logic, where there are multiple possible outcomes based on different combinations of input values. It provides a systematic approach to identifying all the combinations of conditions and also helps to find any gaps or contradictions in the requirements.

This technique is particularly useful when the test object is specified in the form of flowcharts or tables of business rules, when we see statements like “If A and B are true then perform action C”, etc.

Drawbacks

Finding all the interacting conditions can be challenging, particularly when requirements are not well-defined or not documented at all. If there are too many conditions, the number of rules will grow exponentially which can make it challenging to create, read, and maintain the table.

Besides, decision tables do not inherently capture the sequence or temporal aspects of decisions. They focus on logic but do not provide a natural way to represent decisions that depend on the order of events.

Types of Defects

Typical defects include incorrect logic-related processing based on particular combinations of conditions that can result in the following issues:

• The wrong actions happen.
• The right actions don’t happen.
• Conditions interact in an undesirable way.

Conclusion

Test design techniques can be a good tool for requirements analysis and test case development, however, it is necessary to balance test design efforts with priorities and risk levels.

We can invent as many tests as we will not be able to perform, so it is important to care about the efficiency of the tests (their ability to detect serious defects), not about increasing the number of tests.