API Testing Showdown: Postman vs Pytest. Part 2
This part will explore the methods for preparing our system before testing and restoring it afterward, commonly called setup and teardown procedures.
Read “API Testing Showdown: Postman vs Pytest. Part 1” where we consider how these tools cope with entry-level tests, examine the variables system in Postman, and try speculating on how Postman is suitable as an IDE.
Pytest setup/teardown
Fixtures
On the pytest level, we have another marvelous feature — fixtures. Fixtures are functions the framework runs before and/or after test execution. This is like your classic setup/teardown, but the modular way. The simplest example:
def test_add_castle(
basic_url, # this is a fixture call
new_world_id, # and this is too
):
# you can see how basic_url and new_world_id fixtures
# are used in the f-string below
url = f"{basic_url}/world/{new_world_id}/castle"
name = names.get_first_name()
response = requests.post(url, json={"name": name})
assert response.ok
assert name == response.json()["castle"][0]["name"]basic_url and new_world_id are fixtures. They are prepared outside of the test. Let’s go deeper with this example.
The schema
Imagine your customer is Bowser from the Super Mario Bros universe. The supervillain wants us to create an API for managing worlds and castles under his control (app).
POST method test
Let’s get back to the POST example.
def test_add_castle(basic_url, new_world_id):
# assembly
url = f'{basic_url}/world/{new_world_id}/castle'
name = names.get_first_name()
# act
response = requests.post(url, json={'name': name})
# assert
assert response.ok
assert name == response.json()['castle'][0]['name']The test can be broken down into three parts:
- Assembly step: request preparation. This is where our fixtures come into use.
- The act — we send the request.
- And the assert step. You don’t have to involve additional libraries here, as pytest handles regular Python’s assert operator perfectly.
How were the basic_url and new_world_id created?
@pytest.fixture # converts any function into a fixture
def new_world_id(basic_url):
"""Adds a new world, returns its ID, and deletes it when testing is complete"""
# setup
response = requests.post(f"{basic_url}/addworld", json={"name": "whatever"})
worldid = response.json()["world"][0]["id"]
# jump into the test
yield worldid
# teardown
requests.delete(f"{basic_url}/world/{worldid}")Among many things, pytest fixtures might be my favorites. You can easily create them using the @pytest.fixture decorator.
The fixture’s body consists of 3 parts:
- Our setup. In this case, I want to create a new world. So I call POST /addworld and save its worldid.
- Usual Python yield refers to the point where we go to the test. After the test execution, the code after the yield will be called.
- Teardown: we delete the world to avoid cluttering our DB.
As you probably noticed, the new_world_id calls basic_url fixture. This is another marvelous feature: you can chain fixtures.
The bird’s eye view
This is the crucial difference between fixtures and classic setup/teardown. Fixtures allow you to prepare the environment and SUT in a modular way.
By the way, pytest allows you to use regular x-unit setup and teardown functions.
def setup_module(module):
""" setup any state specific to the execution of the given module."""
def teardown_module(module):
""" teardown any state that was previously setup with a setup_module
method.
"""But there is no reason to use them instead of fixtures because they provide more flexibility: you can reuse fixtures in different test modules, combine them for different tests, etc.
Postman setup/teardown
I see the following ways how you can setup/teardown, and, more generally, orchestrate the order of script execution:
Pre-scripts / test scripts (collection/folder/request)
As I mentioned in Part 1, pre-scripts and test scripts are a native way to prepare and clean up data.
This approach is limited:
- Postman scripts are run in strict order. You cannot run a request-level script before collection level one. In pytest, you can simply reorder fixtures in the list.
@pytest.fixture
def fixture1():
print("I'm Fixture 1 !")
@pytest.fixture
def fixture2():
print("I'm Fixture 2 !")
def test_fixture_order(fixture1, fixture2):
# fixture1 goes first
...
def test_fixture_order(fixture2, fixture1):
# now fixture 2 goes first
...
- Postman scripts are run once per request. You cannot determine a script to run once per session or collection. Pytest allows you to do this without friction.
@pytest.fixture (scope="function")
@pytest.fixture (scope="class")
@pytest.fixture (scope="module")
@pytest.fixture (scope="package")
@pytest.fixture (scope="session")- Postman scripts cannot be skipped if needed. If you determine a collection-level script, it will run before each request with no exceptions. And you cannot disable it for a particular request. In the case of Python, you can omit a fixture call in a test if it is not required.
- Postman scripts cannot be altered on the test level. Imagine requesting a prepared user specially designed for a specific case. And you want the user to sign in before running the request. There are plenty of ways to do this with pytest — for example, indirect fixture.
@pytest.fixture
def signed_in_user(request):
login = request.param.get("login")
password = request.param.get("password")
return User(login=login, password=password).sign_in()
@pytest.mark.parametrize(
"signed_in_user", [{"login": "login@mailserver.com", "password": "test_password"}], indirect=True
)
def test_indirect(signed_in_user):
...As the order of scripts in Postman is strict, you cannot easily describe fixture-like functions on the collection level and pass setup parameters on a per-request basis. You can certainly use one of the duct tape solutions, Postman CLI described above, or code in variables — we will discuss this later. They are much less convenient than native pytest fixtures.
- You cannot determine multiple scripts/fixtures on the same level. In pytest, using all available fixtures for each test is not obligatory.
@pytest.fixture
def fixture1():
print("I'm Fixture 1 !")
@pytest.fixture
def fixture2():
print("I'm Fixture 2 !")
def test_single_fixture_used(fixture2):
...
As Postman executes scripts each time you’re trying to make a request, you’re forced to split collections into more folders and spread boilerplate across your project. By the way, pytest allows you to use similar behavior. Just mark a fixture as autouse=True, and it will be executed for each test automatically.
@pytest.fixture(autouse=True)
def fixture3():
print("I'm Fixture 3! And I'm called automatically!")
def test_autouse_fixture():
...
pm.SendRequest
You can send requests straight from Postman scripts. This allows you to prepare and/or clean test data. Thus, you can pack complicated setups/teardowns into a script without distracting yourself with redundant requests.
pm.sendRequest({
url:'http://some_url',
method: 'GET',
header: {
'content-type': 'application/json',
'authorization': request.headers["authorization"]
},
}, function (err, res){
// do something
});Such implicit calls make your tests inconvenient to support and debug. It can be sometimes confusing: you clicked the button “send” just once, but multiple calls were sent. Do I need to say the word “fixture” again?
Request
If you don’t need to cover a complicated scenario, you can proceed with additional requests for setup/teardown. It’s more evident than pm.SendRequest, but you cannot share this setup among collections/directories. And if you need to do multiple requests before an actual check, your setup can get too long to be comfortable to work with.
Requests + setNextRequest
If you have a complicated flow, don’t want to repeat yourself, and want to see all requests clearly, you might want to use another duct tape solution — flow with SetNextRequest.
The idea is simple: using setNextRequest, you can alter the order of requests.
Thus, you can mimic session/folder-scoped fixtures, create loops, reuse code, etc. As a prominent example, I want to refer to the article “My Code Snippets From The London Postman Summit” by Paul Farrell. Sadly, it suggests a brilliant idea that is just a workaround for flaws of Postman requests.
I have translated Paul Farrell’s example to our “castle system” to show the challenges you can face while building a framework using setNextRequest. The whole collection is here.
The structure
The same product: 2 endpoints representing World<->Castle relationship.
Imagine we have a more complicated task:
- Check castle adding
- Check castle editing
- Check getting a single castle
- Check getting a full list of castles
Of course, we can use the “flat” approach described above, but let’s dive into setNextRequest:
Don’t be frustrated; we will break it down into details soon.
We can break it into two main parts. The first one — Before each hooks
This is our setup. The second one is our “tests”, presented as separate folders.
Setup
Remember when I mentioned that you cannot create session-scoped scripts? You can mimic this. Using setNextRequest, you can direct the test runner to a desired script using a counter variable. Thus, you can avoid running a script twice. In this case, a session-scoped script is used for counter initialization only, but you can do everything you want here. You will see below how the counter can help us build a test suite.
(Yes, I know, this looks ugly.)
Before each > Add Castle
This is our kind of “test-scoped fixture”. Its purposes are:
- reset the Postman environment before each test;
- prepare test data (create a new castle);
- Orchestrate the order of test runs. Pay attention, it won’t run before each request, but it will before each folder. We cannot just put this into a pre-request folder because it’ll run each time we call a request inside a folder. But we need this setup to be run just once per folder.
Before each — pre-request
In the first row, we take the current value of the counter. Remember, at the beginning, it is 0; we set this up in the setup request described above.
We want our environment to be clean and ready for the new test run. This is why we call the clear() method…
… and reset the counter. Unfortunately, there is no way to clear everything except the desired variables.
Here, we’ve just initialized a couple of variables required for tests.
Before each — request
After pre-request scripts, the POST castle request happens. We added a new castle using the env vars set in the pre-request.
Before each — test script
Test scripts, as you know, are executed after requests. In this case, the test script is another orchestration part rather than a verificator.
On the very first step, we get the counter. Again. Any script in Postman is run in a separate sandbox. Even pre-/test- scripts of the same request are separated.
You’ve already seen this part of the code. By using setNextRequest, we can change the flow of request execution. The counter can help us build the flow we want. On the very first iteration, we go to the “Add Castle > Get All Castles” — the first request of the “Add Castle” folder. On the second one — the first request of the second folder, “Edit Castle”.
I also want your attention to the fact that renaming requests can get tricky. You can easily forget to rename the next step inside setNextRequest after request renaming. There is no “Refactor->Rename”, the feature I am heavily addicted to in PyCharm. You cannot even search for the text in scripts directly from Postman. In massive refactoring, you’d probably want to export the collection or use another way to work with pure code.
In each (!) last request in a folder
Unfortunately, we cannot altogether avoid using boilerplate code. To properly handle the order of execution, we need to:
Increment the counter. As you remember, you cannot just ++ the variable; you have to read->inc->set on your own.
Problems of a setNextRequest workflow
As you can see, building workflows with setNextRequest is not ideal:
- Requests execution order becomes non-linear and thus non-obvious.
- You cannot run any request at your will. You have to stick to a designated entry point. It’s confusing and much less evident than pytest with its fixtures.
- You have to run tests only via Postman Runner or Newman.
- You cannot quickly randomize the order of folders execution with tests.
- You often have to implement additional variables just for orchestration purposes, like “counter” in the example above. You’re building your low-level framework. Why don’t you just use pytest instead?
- Refactoring can get trickier; you may consider using collections export-import or Postman API.
If you use pytest, you don’t need to worry about such low-level things as setting the order of execution and counting test runs. In the example above, we’ve taken many steps for “test collection”. Pytest is not just a test runner. It’s also a test collector. When you run pytest tests, the following happens.
I want to avoid going deeply into details and emphasize that pytest takes on many routines.
Tip: With the — collect-only flag, you can collect tests and see what and in which order can be executed without actual execution.
Highly convenient for debugging.
Conclusion
Working with fixtures feels like playing with Lego bricks. You can prepare many in different sizes and shapes and then use the ones you need precisely when you need them.
In comparison, setting up a testing system in Postman could be more apparent. You cannot just select what code you want to run before and after a request or suite of requests. Building a complicated workflow with setups and teardowns is tough without continuous copying and pasting code.
In part 3, we will consider a new fancy Postman feature — Postman Flows, try storing and calling code straight from Postman variables, and touch Postman’s templates and built-in modules.
