Introduction

In the fast-paced world of software development, how do you ensure quality when you cannot rely on test automation? In this blog post, I introduce the concept of a Manual Test Pyramid, which addresses the aims of a test automation pyramid in a situation where test automation is not an option. By the end of this post, you’ll understand how to implement a layered manual test strategy to maintain high quality and efficiency.

Challenges with Test Automation

In Story Test Plan Template, I described how teams come together story by story to build test automation for quality software at scale. As a quality engineer starting a recent Microsoft Dynamics CRM (CRM) client engagement, I suggested this approach. The client was very interested in incorporating Continuous Integration and Continuous Delivery (CI/CD) including test automation (Continuously testing and releasing to production in small increments). The development team had mixed reactions: client CRM experts Max and Kiran (not their real names) wanted CI/CD but had never employed it. Non-CRM engineers expected CI/CD with test automation as a matter of course. Other CRM experts suggested a technology-driven approach: first lay down the new workflow, then build out integrations with external systems and refine workflow details and dashboards. The CRM experts were sympathetic to test automation and gave some references for test automation tools that work with CRM, but the workflow was their priority.

Business Constraints

Due to business constraints, the client decided “no new technology (including test automation tools)” for this project, and “just build on the Max and Kiran show” (the tools and methods Max and Kiran had been using for the past four years to bring their CRM system to life). Max and Kiran’s success led to unexpected challenges: they built the system from scratch, working directly with stakeholders across their company. It grew and grew to become an essential part of their company’s operations, central to their bottom line. Their company brought us in to keep them competitive. But, our mandate was to scale without new technology.

I was nonplussed because Automation is one of our quality engineering principles. Amongst my quality engineering colleagues, not building test automation on a project has been described as “soul sucking”. And, “regression death spiral” is a concern in software projects in which there is so much manual testing to do that it takes up resources that would otherwise go into building new features, and, in the extreme, makes it very risky to release new features or even small improvements for the risk of breaking something. On the other hand, I and my non-CRM colleagues noticed that with CRM, the “time to business value” appeared to be much faster with CRM (a low-code development system) than with traditional custom coding in languages like Python, Javascript, or Java. When engineers could produce business value with CRM so efficiently, tying them down with a laborious layer of high-code test automation didn’t feel like the right approach. So, how could we guarantee high velocity with quality, without test automation?

The Manual Test Pyramid

Fortunately, we were able to develop a layered story test plan template based on manual testing.

Layers of Our Manual Test Pyramid

• Story: test each story manually in isolation. Test steps are captured in the story or in a content management system such as Confluence.
• Feature: test each new feature manually once all of the stories that make up the feature have been completed. Test steps are captured in a content management system.
• Product: test the entire product as needed. Test steps are captured in a content management system by engineers and used by stakeholders for user acceptance testing. They are also captured in streamlined form in a test management system such as TestRail by the change management team for efficient regression testing.

Our whole team bought into these layers of testing. Max and Kiran were pleased because it enabled their small team to scale to a large team, while maintaining quality. Our CRM experts reported that “this brought quality to the next level; it really upped our game.” And, when the client’s Change Management team was engaged to manage releases, they found the Product tests to be great for kicking off user acceptance testing by the stakeholders, and to be a great basis for concise manual regression tests in TestRail.

Implementation Example: ExerMart

Although it would be fascinating to do so, I can’t go into more detail on the story with this client. Therefore, in order to make the manual test pyramid concrete, I’m going to extend an example that I started in Story Test Plan Template. There, I introduced the ExerMart application, which allows users to list used exercise equipment for sale, shop for equipment, and compare prices across multiple marketplaces.

ExerMart Backstory

ExerMart has been successful, garnering thousands of users via word of mouth. Most of those users are individuals, but a few are organizations such as gyms that buy, sell, and trade a lot of equipment. Gyms love ExerMart because it provides a low-cost trading platform for the equipment their users love and depend on. And gyms already provide over half of ExerMart’s revenue. We decide to use CRM Sales to build a Gym Workflow that will be used to attract as many gyms as possible to ExerMart.

Workflow Steps

• Generate CRM Lead for a gym.
• Qualify Lead, which becomes a CRM Opportunity.
• Onboard Opportunity to ExerMart (includes service agreement, payment logistics, etc.). Opportunity becomes a CRM Entity. CRM Entity corresponds to a user in ExerMart.

ExerMart Enhancement

Add support for bank payments.

Updated Architecture

Backlog

To illustrate the growth of our manual test pyramid, we’ll build the work described in the following backlog of stories. Our main focus will be on CRM and manual testing, whereas, the ExerMart work is provided for context.

• Epic/Feature 1: Create CRM workflow (aka Business Process Flow or BPF)
  - Story 1: Create initial BPF
  - Story 2: Generate Lead
  - Story 3: Qualify Lead to Opportunity
  - Story 4: Onboard Opportunity
• Epic/Feature 2: Enhance ExerMart for Bank Payments
  - Story 5: Add bank payment support to Database
  - Story 6: Add bank payment support to API
  - Story 7: Add bank payment support to UI
• Epic/Feature 3: Integrate Onboard Opportunity from CRM to ExerMart
  - Story 8: Collect bank payment information in CRM
  - Story 9: Set up Gyms in ExerMart automatically from CRM

Sprint Overview

We’ll build the work described in the backlog over three sprints. Notice how the manual tests build up layer by later, starting with story tests only in sprint 1, more story tests and a feature test in sprint 2, and still more story and feature tests, as well as an enhanced product test, in sprint 3.

Sprint 1: Getting Started

In sprint 1, we create the skeleton of the workflow in CRM, and begin to lay the supporting groundwork in ExerMart.

Epic/Feature 1: Create CRM workflow
— — Story 1: Create initial BPF

• Story Test: manually run through the business process flow. Make sure that empty Lead and Opportunity are created and that the final state is Opportunity Onboarding.
• Feature Test: Nothing to test until the feature is complete.
• Product Test: Nothing new to test until CRM is integrated with ExerMart.

Epic/Feature 2: Enhance ExerMart for Bank Payments (automated testing)
— — Story 5: Add bank payment support to Database

User Acceptance Testing: Nothing to test yet

Sprint 2: Put all the supporting pieces in place

In sprint 2, we put all of the supporting pieces into place. These pieces will be woven together into the product in sprint 3.

Epic/Feature 1: Create CRM workflow
— — Story 2: Generate Lead
— — Story 3: Qualify Lead to Opportunity
— — Story 4: Onboard Opportunity

• Story Test: manually verify the details separately for Generate Lead, Qualify Lead to Opportunity, and Onboard Opportunity.
• Feature Test: manually verify the flow through from Generate Lead to Onboard Opportunity
• Product Test: Nothing new to test until CRM is integrated with ExerMart.

Epic/Feature 2: Enhance ExerMart for Bank Payments (automated testing)
— — Story 6: Add bank payment support to API
— — Story 7: Add bank payment support to UI

User Acceptance Testing: Nothing to test yet

Sprint 3: Tie it all together

In sprint 3, we tie all the pieces from sprints 1 and 2 together.

Epic/Feature 3: Integrate Onboard Opportunity with ExerMart
— — Story 8: Collect bank payment information in CRM
— — Story 9: Set up Gyms in ExerMart automatically

• Story Test: manually verify collection of bank payment information in CRM.
• Feature Test: manually verify setup of Gyms in ExerMart via the CRM system.
• Product Tests:
  - manual end to end test — gym lead, qualified lead, onboarded gym in ExerMart, transactions in ExerMart

User Acceptance Testing: Follow the manual Product Tests.

Reflections and Optimization

Let’s recap:

• Sprint 1: One story test is added.
• Sprint 2: Two additional story tests and one feature test are added.
• Sprint 3: One additional story test, a second feature test is added, and a product test are added.

Our manual test pyramid looks like this:

If only building ExerMart with a gym onboarding workflow were this easy! (Wait, could it be this easy? Ummm, ChatGPT, I have an opportunity for you…)

One of the main concerns with building manual test cases instead of automated tests is the danger of a regression death spiral. In order to combat this, Max and Kiran’s Change Management team kept a highly-optimized manual test suite built out of the manual Product Tests. After 12 sprints, they had 200 tests which took an engineer about 8 hours to execute. Aware of the dangers, they prioritized and optimized the suite sprint by sprint. Prioritization enabled them to execute the highest value tests first, finding the most important bugs early in the test pass. Optimization enabled them to consolidate redundant steps and make use of common setups. For example, many details of gym onboarding could be tested with the set up of of just one gym. Optimization often involves tradeoffs such as less complete test coverage of all combinations, so these decisions should be made carefully, considering risk of bugs vs. time to execute tests. A regression death spiral is not an inevitable result of manual testing. Regression test suites and overhead must be carefully managed whether testing is manual or automated.

Key Takeaways

• Maintaining a prioritized and optimized manual test suite can prevent or mitigate a regression death spiral.
• Careful management of regression test suites is crucial, whether tests are manual or automated.

Conclusion

There are times when manual testing is the primary mode of testing. Nonetheless, the principles outlined in Story Test Plan Template still apply: create tests story by story as you go, with low-level tests feeding into medium-sized tests, and up to tests at the level of the entire project. Here is a comparison of our manual test layers with the automated layers from Story Test Plan Template.

Whether manual, automated, or mixed, we want layers of tests, built up story by story.

References

• Microsoft Dynamics CRM
• Quality Engineering Core Principles
• Story Test Plan Template
• TestRail Test Management System
• The Regression Death Spiral

Thanks

• Max and Kiran and your colleagues. You know who you are. You inspire me.
• My project team at Slalom that built on the Max and Kiran show.
• Slalom Chat Beta based on GPT-4o, for feedback on the first and second drafts.
• Jeff Averill for feedback.