Implementing AI in Software Testing: Creating a Text Generation Model for Test Automation

Dr. Ernesto Lee

·

Follow

7 min read

·

Feb 6, 2024

--

5

Listen

Share

Introduction

Incorporating Artificial Intelligence (AI) in software testing is a game-changer, offering significant efficiency and effectiveness boosts. This article utilizes OpenAI’s text generation models — particularly GPT-3.5-turbo and GPT-4-turbo-preview — to craft a Text Generation Model in Google Colab, focusing on a Test Automation use case.

Use Case: Automated Test Case Generation

Our use case revolves around automating test case generation for software applications. This process, traditionally manual and labor-intensive, can be streamlined significantly by employing a Text Generation Model to automatically generate test scenarios from user stories or requirements.

Steps to Build the Model

Step 1: Environment Setup

Start by setting up your environment in Google Colab, a free Jupyter notebook environment provided by Google.

1. Open Google Colab and create a new notebook.
2. Install the OpenAI package:

!pip install openai -q

Step 2: Import Libraries

In your notebook, import the necessary libraries:

from openai import OpenAI

Step 3: OpenAI Authentication

Acquire your OpenAI API key and authenticate as shown:

# openai.api_key = 'your-api-key'
client = OpenAI(api_key='your-api-key')

Step 4: Defining the Test Case Generation Function

Define a function to generate test cases based on software requirements.

def generate_test_cases(requirement):
    response = client.chat.completions.create(
      model="gpt-3.5-turbo",
      messages=[
        {"role": "system", "content": "You are a helpful assistant capable of generating software test cases."},
        {"role": "user", "content": requirement}
      ]
    )
    return response.choices[0].message.content

Step 5: Test the Function

Test the function with a sample requirement:

requirement = "The system shall allow users to securely login with a username and password."
test_cases = generate_test_cases(requirement)
print(test_cases)

Step 6: Output Analysis and Refinement

Evaluate the output for its relevance and completeness, refining the prompt or parameters as needed.

Step 7: Integration with Test Management Tools

Optionally, integrate the output with test management tools or repositories to automate adding new test cases to your suite.

Conclusion

You’ve now created a tool to generate test cases using OpenAI’s text generation models. This tool not only saves time but also ensures a level of consistency and thoroughness difficult to achieve manually.

Future Enhancements

• Integrate GPT-4-vision-preview for GUI testing.
• Implement reproducible outputs for consistency.
• Use JSON mode for structured output compatible with test management tools.

Staying adaptive and exploratory is crucial in the dynamic field of AI in software testing, unlocking its full potential.

BONUS: UPDATE THE AND CREATE NEW FUNCTIONS FOR OTHER AREAS OF TEST AUTOMATION

Use Case 2: Generating Regression Test Scenarios for a Shopping Cart Feature

Objective: Automate the generation of regression test scenarios for a shopping cart feature in an e-commerce application to ensure new changes don’t break existing functionalities.

Code Walkthrough:

1. Set Up Authentication:

• Use your OpenAI API key to authenticate with the OpenAI client.

from openai import OpenAI
client = OpenAI(api_key='your-api-key')

Define the Test Case Generator Function:

• This function will take a feature description and return regression test scenarios.

def generate_regression_tests(feature_description):
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a helpful assistant capable of generating regression test scenarios for a shopping cart feature."},
            {"role": "user", "content": feature_description}
        ]
    )
    return response.choices[0].message.content

Invoke the Function with Feature Description:

• Provide a description of the shopping cart feature to generate test scenarios.

feature_description = "Ensure that the shopping cart allows users to add items, remove items, and proceed to checkout."
regression_tests = generate_regression_tests(feature_description)
print(regression_tests)

Evaluate and Refine:

• Analyze the generated test scenarios and iterate on the prompt if needed to ensure comprehensive coverage.

Use Case 3: Validating API Responses for a Weather Forecasting Service

Objective: Generate test cases to validate JSON API responses for a weather forecasting service, ensuring the data structure and values are as expected.

Code Walkthrough:

1. Set Up Authentication:

• Authenticate with the OpenAI API using your provided API key.

from openai import OpenAI
client = OpenAI(api_key='your-api-key')

Define the Test Case Generator Function:

• This function will take an API endpoint description and return test cases to validate the API’s JSON response.

def generate_api_validation_tests(api_description):
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a helpful assistant that generates test cases to validate JSON responses from an API."},
            {"role": "user", "content": api_description}
        ]
    )
    return response.choices[0].message.content

Invoke the Function with API Description:

• Provide a description of the API endpoint to generate validation test cases.

api_description = "The weather API should return a JSON response with fields for temperature, humidity, and precipitation forecast for the next 5 days."
api_validation_tests = generate_api_validation_tests(api_description)
print(api_validation_tests)

1. Evaluate and Refine:

• Review the generated test cases for accuracy and completeness. Make sure the test cases check for the presence of each field and the correctness of the data format.

User Instructions:

• Ensure you replace 'your-api-key' with your actual OpenAI API key.
• Run each code block in a Jupyter notebook environment, such as Google Colab.
• After running the test case generator functions, review the suggested test cases.
• If the output is not satisfactory, refine the feature or API descriptions to be more specific or adjust the system message to guide the model better.
• Iterate on the inputs and system message until the generated test cases meet your requirements for coverage and detail.

By following these examples, you can expand the capabilities of your AI-driven test generation to cover various aspects of software testing, making your testing process more robust and efficient.

Let’s create a Web App for this

Step 1: Create a GitHub Account

1. Go to GitHub’s website.
2. Click on the “Sign up” button in the top-right corner.
3. Fill in the required fields with your username, email address, and password for your new GitHub account.
4. Verify your account through the email that GitHub sends you.
5. Complete the setup by following the on-screen instructions.

Step 2: Create a New Repository

1. Once logged in, click on the “+” icon in the top-right corner and select “New repository.”
2. Give your repository a name, such as “streamlit-test-case-generator.”
3. Choose if you want your repository to be public or private.
4. Initialize the repository with a README file.
5. Click “Create repository.”

Step 3: Add Files to Your Repository

1. In your repository, click on “Add file” and select “Create new file.”
2. Create a file named app.py—this will be your main Python file for the Streamlit app.
3. Write your Streamlit code into app.py. Ensure your code includes error handling for the API key to avoid exposing it.
4. Create another file named requirements.txt. This file should list all Python libraries that your app depends on, including streamlit and openai.
5. Commit the new files by clicking “Commit new file.”

Here is app.py:

import streamlit as st
import openai
import os

# Retrieve the API key from the environment variable
OPENAI_API_KEY = os.getenv('OPENAI_API_KEY')

# Initialize the OpenAI client with the API key
openai.api_key = OPENAI_API_KEY

# Define the function to generate test cases
def generate_test_cases(requirement):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a helpful assistant capable of generating software test cases."},
            {"role": "user", "content": requirement}
        ]
    )
    return response.choices[0].message.content

# Streamlit app layout
st.title('AI-powered Test Case Generator')
st.write('Enter your software requirement to generate test cases.')

# Text area for user to enter the software requirement
requirement = st.text_area("Requirement", height=150)

# Button to generate test cases
if st.button('Generate Test Cases'):
    if requirement:
        with st.spinner('Generating...'):
            try:
                test_cases = generate_test_cases(requirement)
                st.success('Generated Test Cases')
                st.write(test_cases)
            except Exception as e:
                st.error('An error occurred while generating test cases.')
                st.error(e)
    else:
        st.error('Please enter a requirement to generate test cases.')

Here is requirements.txt

streamlit
openai

Step 4: Add Your API Key as a Secret

1. Go to your GitHub repository’s Settings tab.
2. Find the “Secrets” section in the left sidebar and click on “Actions.”
3. Click on “New repository secret.”
4. Name your secret (e.g., OPENAI_API_KEY) and paste your OpenAI API key as the value.
5. Click “Add secret” to save it.

Step 5: Create a Share.streamlit.com Account

1. Visit share.streamlit.com and click on “Sign up.”
2. Sign up using your GitHub account to link your Streamlit account with GitHub.

Step 6: Deploy Your Streamlit App

1. Once logged into Streamlit, click on “New app.”
2. Select the GitHub repository you created earlier.
3. Choose the branch where your files are located (usually main or master).
4. Write app.py in the "Path to a Streamlit app" field.
5. In the “Advanced settings,” input your secret key (OPENAI_API_KEY) into the "Environment variables" section.
6. Click “Deploy” to deploy your app. Streamlit will automatically install the dependencies listed in your requirements.txt file and deploy your app.