Paul Grizzaffi

You’ve decided that test automation is working or is going to work for you. That’s good; you’ve done your homework. Furthermore, you’ve decided that having a common or “base” automation team is your preferred organizational approach; you’ve decided this team will build and support the common automation infrastructure that will be shared across the teams in your organization or your company. Awesome!

How are you going to fund that team?

Presented to:
STPCon Fall 2020

We just finished automating “all” of our regression test suite. Or, perhaps, we now have 4731 BDD “test scripts” that we run on each deployment. Hooray! T-shirts for everyone! Then it happens…our product changes, our tool version changes, or our infrastructure changes. Our shiny new toy becomes a tarnished, broken, and unusable pile of shattered dreams and wasted effort.

Having automation is reminiscent of owning a cast iron pot. Like cast iron… Show more

Presented to:
STPCon Fall 2020

We just finished automating “all” of our regression test suite. Or, perhaps, we now have 4731 BDD “test scripts” that we run on each deployment. Hooray! T-shirts for everyone! Then it happens…our product changes, our tool version changes, or our infrastructure changes. Our shiny new toy becomes a tarnished, broken, and unusable pile of shattered dreams and wasted effort.

Having automation is reminiscent of owning a cast iron pot. Like cast iron cookware, and like the software for our products, our automation software requires proper care and feeding; typically, we call this maintenance. If cast iron cookware is not properly maintained, especially when it’s new, it will corrode, rust, and deteriorate to the point that it’s no longer fit for purpose. Considerable effort will be expended to restore the cookware’s usability; the same is true for automation.

In this session, Paul Grizzaffi will explore causes of automation deterioration and explain ways we can prevent our automation’s value from rusting away. Show less

I will not be presenting THE Ten Commandments of Test Automation. Instead, I’ll be presenting Ten MORE Commandments of Test Automation. And away we go…

The drive-thru is but one way to organize teams for automation; there are, however, three ways: the dedicated automation team, the distributed automation team, and the hybrid approach.

Let’s explore!

My keynote for the Odyssey conference.

When these script failures occurred, the status would unfailingly be “we could not get the scripts to pass”.

I knew what they meant, but the wording bugged me. I let that go on for a few weeks because I thought maybe it was just me. Perhaps, I was being too sensitive to the vocabulary; I wasn’t.

The key here is value: Is expending effort on a specific automation activity the right thing to do, and is this it the right time to do it? Answering these questions will help you quantify the value. Calculating the opportunity cost and the total cost of ownership of both the automation and the application itself are essential parts to answering those questions.

Much is made of the dream flow from continuous integration to continuous testing to continuous delivery to production. Conceptually this sounds great! A developer checks in their code, the code is built, unit tests are executed, the code is deployed to a testing environment, and then the automated test scripts are executed.

What could possibly go wrong? What, indeed!

As a QA, QE, or testing professional, you hear some questions frequently, especially if you are in a leadership role. These include: How many test cases are left to execute, how much longer will testing take, and what percentage of our testing is complete?
Co-authored with Mas Kono.

So, as a test engineer, how do you track down your own elusive Sasquatch? I use an approach I call "periodic automation," and it works quite well.

To borrow from a popular Internet meme, one does not simply walk into test automation and parallelize. As with most things, especially software things, there are many considerations to address beforehand. Here are a few of them.

Presented to:
STPCon Spring and Fall 2019, Ministry of Testing Dallas, Test Bash SF 2019

Arranging the playing cards in a deck to be in one’s favor is called stacking the deck. Outside of card playing, we use the term more generally to mean arranging a situation to increase our chances of a favorable outcome. When it comes to automation endeavors, the meaning is no different. Specifically, we want to arrange our architecture, implementation, and usage patterns to be appropriate for… Show more

Presented to:
STPCon Spring and Fall 2019, Ministry of Testing Dallas, Test Bash SF 2019

Arranging the playing cards in a deck to be in one’s favor is called stacking the deck. Outside of card playing, we use the term more generally to mean arranging a situation to increase our chances of a favorable outcome. When it comes to automation endeavors, the meaning is no different. Specifically, we want to arrange our architecture, implementation, and usage patterns to be appropriate for our endeavor’s desired life-span.

One approach to future-proofing is to focus less on the automation framework and more on the automation stack. An automation stack is a layered automation architecture where each layer builds upon the previous one and provides an audience-appropriate interface to the lower levels’ capabilities. This layered approach helps extend an implementation’s longevity by increasing the portability of the implementation across frameworks and across tools.

Join Paul Grizzaffi as he walks through how layers can be a valuable part of an automation implementation, some caveats that should be considered when layering an architecture, and several examples of layered architectures of which he’s been a part. Come and learn ways to stack the deck in YOUR favor.
Show less

Presented to:
Automation Guild 2019

We just finished automating “all” of our regression test suite. Or, perhaps, we now have 4731 BDD “test scripts” that we run on each deployment. Hooray! T-shirts for everyone! Then it happens…our product changes, our tool version changes, or our infrastructure changes. Our shiny new toy becomes a tarnished, broken, and unusable pile of shattered dreams and wasted effort.

Having automation is reminiscent of owning a cast iron pot. Like cast… Show more

Presented to:
Automation Guild 2019

We just finished automating “all” of our regression test suite. Or, perhaps, we now have 4731 BDD “test scripts” that we run on each deployment. Hooray! T-shirts for everyone! Then it happens…our product changes, our tool version changes, or our infrastructure changes. Our shiny new toy becomes a tarnished, broken, and unusable pile of shattered dreams and wasted effort.

Having automation is reminiscent of owning a cast iron pot. Like cast iron cookware, and like the software for our products, our automation software requires proper care and feeding; typically, we call this maintenance. If cast iron cookware is not properly maintained, especially when it’s new, it will corrode, rust, and deteriorate to the point that it’s no longer fit for purpose. Considerable effort will be expended to restore the cookware’s usability; the same is true for automation.

In this session, Paul Grizzaffi will explore causes of automation deterioration and explain ways we can prevent our automation’s value from rusting away.
Show less

As with your vintage truck, your carefully crafted automation, if not properly maintained, will corrode and rust to the point that it's no longer fit for its purpose—especially when that automation is new. If that happens, you'll have to expend considerable effort to restore usability.

Presented to:
TestBash USA 2018

Breaking news! Automation development is software development. Yeah, it’s true. Even if we are using a drag-and-drop or record-and-playback interface to create that automation, somewhere, in the stack, under the hood or behind the curtain, there is code sequenced by our actions. We must start treating our automation initiatives as software development initiatives, lest we end up in a quagmire of unsustainability and early project… Show more

Presented to:
TestBash USA 2018

Breaking news! Automation development is software development. Yeah, it’s true. Even if we are using a drag-and-drop or record-and-playback interface to create that automation, somewhere, in the stack, under the hood or behind the curtain, there is code sequenced by our actions. We must start treating our automation initiatives as software development initiatives, lest we end up in a quagmire of unsustainability and early project death.

Automation activities that aren’t treated as software activities run the risk of being underestimated, delivered late, and being difficult to maintain; each of these scenarios take a bite out of our budget. Join Paul Grizzaffi as explains why automation really is software and the key points of software development that we should keep in mind when creating automation software, such as encapsulation, abstraction, DRY, and YAGNI.
Show less

Presented To:
STAREast 2018 (keynote), OnlineTestConf 2018 (keynote), QA Trailblazers Exchange

The phrase "test your mettle" means to test what one is made of, what one is prepared to endure. In both testing and automation development, we often have to endure suboptimal situations and difficult challenges such as unrealistic expectations, resistance to change, and an insistence that automation has to come from test cases.

In coping with these challenges, many people find solace… Show more

Presented To:
STAREast 2018 (keynote), OnlineTestConf 2018 (keynote), QA Trailblazers Exchange

The phrase "test your mettle" means to test what one is made of, what one is prepared to endure. In both testing and automation development, we often have to endure suboptimal situations and difficult challenges such as unrealistic expectations, resistance to change, and an insistence that automation has to come from test cases.

In coping with these challenges, many people find solace in music; listening to and playing music can be a cathartic experience. Music, specifically titles and lyrics, can also be thought-provoking and motivating. Paul Grizzaffi, a self-proclaimed metalhead and rock-a-holic, is one of those people who is greatly affected by music. He not only has experience with testing and automation, he’s quite the aficionado of heavy music. His love of automation, testing, and music inspired the pun in the title.

Join us as Paul takes us on a tour of some impactful lyrics and song titles that frame memorable messages about testing and automation. Among the featured messages: we make our own expectations, automation is programming, not magic, and there are important business aspects related to automation and testing that cannot be ignored.
Show less

Presented to:
STAREast 2018 (keynote), OnlineTestingConf 2018 (keynote), QA or the Highway 2020 (keynote)

The phrase "test your mettle" means to test what one is made of, what one is prepared to endure. In both testing and automation development, we often have to endure suboptimal situations and difficult challenges such as unrealistic expectations, resistance to change, and an insistence that automation has to come from test cases.

In coping with these challenges, many people… Show more

Presented to:
STAREast 2018 (keynote), OnlineTestingConf 2018 (keynote), QA or the Highway 2020 (keynote)

The phrase "test your mettle" means to test what one is made of, what one is prepared to endure. In both testing and automation development, we often have to endure suboptimal situations and difficult challenges such as unrealistic expectations, resistance to change, and an insistence that automation has to come from test cases.

In coping with these challenges, many people find solace in music; listening to and playing music can be a cathartic experience. Music, specifically titles and lyrics, can also be thought-provoking and motivating. Paul Grizzaffi, a self-proclaimed metalhead and rock-a-holic, is one of those people who is greatly affected by music. He not only has experience with testing and automation, he’s quite the aficionado of heavy music. His love of automation, testing, and music inspired the pun in the title.

Join us as Paul takes us on a tour of some impactful lyrics and song titles that frame memorable messages about testing and automation. Among the featured messages: we make our own expectations, automation is programming, not magic, and there are important business aspects related to automation and testing that cannot be ignored.
Show less

Presented to :
SauceCon 2018


Roughly speaking, ‘fuzzing’ is testing without an oracle, i.e. testing without knowing what a specific outcome should be. When fuzzing, we don’t necessarily know what should happen but we have a good idea of some things that shouldn’t happen, such as 404 errors and server or application crashes. We generally apply fuzzing to produce these kinds of errors when we’re testing text boxes, but why should textboxes have all the fun?

Websites created… Show more

Presented to :
SauceCon 2018


Roughly speaking, ‘fuzzing’ is testing without an oracle, i.e. testing without knowing what a specific outcome should be. When fuzzing, we don’t necessarily know what should happen but we have a good idea of some things that shouldn’t happen, such as 404 errors and server or application crashes. We generally apply fuzzing to produce these kinds of errors when we’re testing text boxes, but why should textboxes have all the fun?

Websites created today are highly interconnected, multi-server applications that include connections to out-of-network servers that are controlled by neither the applications nor the team. This situation makes it difficult to both enumerate and control all the possible combinations of paths through our system. Even if we could identify all the possible paths, most organizations would not have the time to test all of these scenarios, regardless of whether or not they apply automation to help with that testing.

During this session, we explore how expanding our automation approach by using randomization can help mitigate the risks associated with hard-to-enumerate application scenarios. By using random clicking, we can provide testers with additional information via exploring paths through the application which are not intuitive, but which are still valid. We’ll discuss why creating a random clicker doesn’t have to take a lot of effort, how this approach is rooted in the facets of High Volume Automated Testing (HiVAT), and some considerations of which to be mindful when using randomization.
Show less

Presented to:
STPCon Fall 2018, STAREast Spring 2019, SauceCon 2018

Roughly speaking, ‘fuzzing’ is testing without an oracle, i.e. testing without knowing what a specific outcome should be. When fuzzing, we don’t necessarily know what should happen, but we have a good idea of some things that shouldn’t happen, such as 404 errors and server or application crashes. We generally apply fuzzing to help uncover these kinds of errors when we’re testing text boxes, but why should textboxes… Show more

Presented to:
STPCon Fall 2018, STAREast Spring 2019, SauceCon 2018

Roughly speaking, ‘fuzzing’ is testing without an oracle, i.e. testing without knowing what a specific outcome should be. When fuzzing, we don’t necessarily know what should happen, but we have a good idea of some things that shouldn’t happen, such as 404 errors and server or application crashes. We generally apply fuzzing to help uncover these kinds of errors when we’re testing text boxes, but why should textboxes have all the fun?

Websites created today are highly interconnected, multi-server applications that include connections to out-of-network servers that are controlled by neither the applications nor the team. This situation makes it difficult to both enumerate and control all the possible combinations of paths through our system. Even if we could identify all the possible paths, most organizations would not have the time to test all these scenarios, regardless of whether they apply automation to help with that testing.

During this session, we explore how expanding our automation approach by using randomization can help mitigate the risks associated with hard-to-enumerate application scenarios. By using random clicking, we can provide testers with additional information via exploring paths through the application which are not intuitive, but which are still valid. We’ll discuss why creating a random clicker doesn’t have to take a lot of effort, how this approach is rooted in the facets of High Volume Automated Testing (HiVAT), and some considerations of which to be mindful when using randomization.
Show less

Presented to:
STPCon Fall 2017 (Keynote), InflectraCon 2020 (Keynote)

History has passed judgement on Pyrrhus of Epirus, cementing his legacy with a term based on his name, the definition of which casts Pyrrhus as irresponsible. With the passage of time and the value of hindsight, we are able to make our own judgements as to whether or not he was acting responsibly. We can make similar judgements on other figures from our history and from our own careers.

Being responsible… Show more

Presented to:
STPCon Fall 2017 (Keynote), InflectraCon 2020 (Keynote)

History has passed judgement on Pyrrhus of Epirus, cementing his legacy with a term based on his name, the definition of which casts Pyrrhus as irresponsible. With the passage of time and the value of hindsight, we are able to make our own judgements as to whether or not he was acting responsibly. We can make similar judgements on other figures from our history and from our own careers.

Being responsible means exercising good judgement. Despite this, however, we often eschew good judgement and embark on quixotic quests for “complete, end-to-end automation” or “automate at all cost”; these quests distract us from providing value while we build an unsustainable Rube Goldberg machine of automation sadness. These are not responsible approaches. It is incumbent on us to undertake automation initiatives in a responsible, value-based way. This value can take many forms, but in order to capitalize on that value we must be aware of many factors that affect it.

Join us as Paul Grizzaffi explains responsible ways to approach automation, describes some of the knowledge we’ll need in order to be responsible, and shares insights about automation responsibility from his own career. Let’s allow history to remember our automation initiatives fondly instead of as Pyrrhic forays into irresponsibility.
Show less

Presented to:
QA.Improving.us Meetup, STPCon Spring 2018


Breaking! Automation development is software development. Yeah, it’s true. Even if we are using a drag-and-drop or record-and-playback interface to create automation, somewhere in the stack, under the hood or behind the curtain, there is code that was sequenced by our actions. We must start treating our automation initiatives as software development initiatives, lest we wind up in a quagmire of unsustainability.

One… Show more

Presented to:
QA.Improving.us Meetup, STPCon Spring 2018


Breaking! Automation development is software development. Yeah, it’s true. Even if we are using a drag-and-drop or record-and-playback interface to create automation, somewhere in the stack, under the hood or behind the curtain, there is code that was sequenced by our actions. We must start treating our automation initiatives as software development initiatives, lest we wind up in a quagmire of unsustainability.

One beneficial tactic used in software development is to have a team member go through newly written or modified code, looking for issues and risks; this is called a code inspection or code review. Much as a proofreader or editor will provide feedback on a book or article, code inspectors provide feedback on areas of the code that may benefit from rework, such as supportability, readability, extensibility, and issues or risks of issues.

Join us as Paul Grizzaffi explains why we should do code inspections of our automation software, how inspections for automation can differ from those for product software, and actual issues found during these reviews.
Show less

Presented to:
DFW ASEE Annual Conference 2017 (Keynote)

Synopsis:
In our jobs, as in our lives, we have certain responsibilities to which we must attend, things we have a fiduciary duty upon which to execute. It is no different when addressing test automation. Too often, however, our automation efforts are an “all or nothing” or “at all cost” endeavor; these are not responsible approaches. It is incumbent on us to undertake automation initiatives in a responsible, value-based way… Show more

Presented to:
DFW ASEE Annual Conference 2017 (Keynote)

Synopsis:
In our jobs, as in our lives, we have certain responsibilities to which we must attend, things we have a fiduciary duty upon which to execute. It is no different when addressing test automation. Too often, however, our automation efforts are an “all or nothing” or “at all cost” endeavor; these are not responsible approaches. It is incumbent on us to undertake automation initiatives in a responsible, value-based way. This value can take many forms, but in order to capitalize on that value we must be aware of many factors that affect it.

Join us as Paul Grizzaffi explains responsible ways to approach automation and shares insights about automation from his own career.
Show less

Many automation tools have a mechanism for storing data used in their test scripts. Typically, the specifics of this mechanism is different across tools, making it difficult to use this data outside the tool itself. Using an external, reusable data source allows organizations to avoid the cost of migrating or duplicating existing data, thereby future-proofing their frameworks.

#5 Most Popular Article of 2016 for StickyMinds.

Traditionally, automated scripts are derived from existing test cases. But if we divorce the notion of “automation” from the notions of “test cases” and “test scripts,” we can think of automation as a judicious use of technology to help humans do their jobs. This broadens our world to include different tools that can help testers increase coverage, test faster, and detect trends.

Though automation is often mentioned in the same breath as behavior-driven development, they are not equally important. If you want to use behavior-driven development, do just that: Work on getting the approach right, and forget about the automation at first.

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, STPCon Spring 2015, STPCon Fall 2016

Synopsis:
Most software companies have a test automation initiative; some are just starting, some are humming right along. Typically, these initiatives focus “traditional automation”: using a software package to automate existing test cases or user stories. Not that there’s anything wrong with that; often, there is value in that. But, if we use only the traditional approach to automation,… Show more

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, STPCon Spring 2015, STPCon Fall 2016

Synopsis:
Most software companies have a test automation initiative; some are just starting, some are humming right along. Typically, these initiatives focus “traditional automation”: using a software package to automate existing test cases or user stories. Not that there’s anything wrong with that; often, there is value in that. But, if we use only the traditional approach to automation, we miss opportunities to exploit the power of automation.

Instead of focusing on how many test cases we can automate, what if we consider this: how can automation help me do my job”?

This session shares some real-world examples where non-traditional automation (that we call “automation assist”) provided business value, including cases where traditional automation could not have provided the same level of value. We'll also discuss some things to keep in mind when implementing automation assist, including automating with tools that are not traditionally considered automation tools.
Show less

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, DevOps Live! Annual Conference, STPCon Spring 2016, DFW Scrum 2017, DevOpsDays DFW 2017, TestBash Philly 2017

Synopsis:
In the pop culture, Sasquatch (also known as Bigfoot) is a possibly non-existent, ape-like creature infrequently seen in the Pacific Northwest. In the software realm, we have our own versions of Sasquatch: those irritating, "intermittent issues". These kinds of issues are typically difficult to find and… Show more

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, DevOps Live! Annual Conference, STPCon Spring 2016, DFW Scrum 2017, DevOpsDays DFW 2017, TestBash Philly 2017

Synopsis:
In the pop culture, Sasquatch (also known as Bigfoot) is a possibly non-existent, ape-like creature infrequently seen in the Pacific Northwest. In the software realm, we have our own versions of Sasquatch: those irritating, "intermittent issues". These kinds of issues are typically difficult to find and are often blamed on anything other than a product defect.

Using a real-world example, Paul Grizzaffi will explain how "periodic automation" can help hunt down these elusive targets. To give additional context, he will explain how this approach is related to High-Volume Automated Testing (HiVAT), including some HiVAT basics and some examples. He will also explain some considerations of which we need to be mindful when implementing periodic automation so as not to become desensitized to failures.
Show less

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, STPCon Spring 2014, STPCon Fall 2015

Synopsis:
What’s the best automation tool? That’s a valid question, right? Right! You've asked it; I've asked it. The question itself is not the concern, but when the question is asked. Too often, this is the first question asked when starting an automation initiative, but is there a more appropriate time to ask it. Let’s discuss the prerequisites to asking this question, work to decide… Show more

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, STPCon Spring 2014, STPCon Fall 2015

Synopsis:
What’s the best automation tool? That’s a valid question, right? Right! You've asked it; I've asked it. The question itself is not the concern, but when the question is asked. Too often, this is the first question asked when starting an automation initiative, but is there a more appropriate time to ask it. Let’s discuss the prerequisites to asking this question, work to decide when it’s time to ask it, and introduce “most appropriate in class”. Show less

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, Mid-Cities DevOps, DFW ASEE 2013 Annual Conference, STPCon Spring 2013, DFW ASEE monthly, DFW Scrum, ICSQ 2014.

Synopsis:
Sailing the treacherous waters of test automation is challenging at best. From executives thinking "it's going to fix all of our quality problems, right?" to team members saying "no, that's YOUR job", our automation initiatives face an uphill battle with a tendency to struggle…or at least to not achieve… Show more

Presented to:
QA Trailblazers Knowledge Exchange, DFWQAA, Mid-Cities DevOps, DFW ASEE 2013 Annual Conference, STPCon Spring 2013, DFW ASEE monthly, DFW Scrum, ICSQ 2014.

Synopsis:
Sailing the treacherous waters of test automation is challenging at best. From executives thinking "it's going to fix all of our quality problems, right?" to team members saying "no, that's YOUR job", our automation initiatives face an uphill battle with a tendency to struggle…or at least to not achieve their full potential. Let's shed some light on several of these challenges and pitfalls while discussing some of the approaches we can use to avoid a voyage on the SS Failboat. Show less