API Testing Showdown: Postman vs Pytest. Part 1

Published in

Exness Tech Blog

10 min readOct 24, 2023

When it comes to API auto-testing, many tools are lying around. Pytest framework coupled with requests library is one of the most popular options nowadays. At the same time, we have a prevalent HTTP-client / API development platform — Postman. And it even has auto-testing capabilities — thanks to the bundled JS engine. But is Postman any good for serious work?

This is part 1 of the series. Here, we will consider how these tools cope with entry-level tests, examine the variables system in Postman, and try speculating on how Postman is suitable as an IDE.

What is Postman?

In the blue corner of the ring, we have Postman.

Postman is a whole API development platform containing a lot of features.

It allows you to work through all necessary parts of API development, including documenting, prototyping, and testing.

The last part — testing — is the thing I want to talk about.

What is Pytest?

In the red corner of the ring, we have pytest. Pytest is a python full-featured general-purpose testing framework. It offers everything you can expect from this type of tool: test runner, marks, fixtures, hooks, etc.

It’s not a visual tool, so no screenshot here.

Out of the box, it’s only suitable for unit testing. To make it possible to test REST API, we need something to make HTTP requests. We will use the requests library, possibly the most popular add-on, for this purpose.

Small remark: we will discuss REST API further down the text, but the same principles apply to other protocols.

The simplest test

Postman

Postman is great as an HTTP client. You don’t need to write a single row of code to see a response.

Of course, you have to write code to make assertions. Postman uses a JavaScript engine for this.

pm.test("Status code is 200", function () {
    pm.response.to.have.status(200);
});
pm.test("2+2=4", function () {
    var jsonData = pm.response.json();
    pm.expect(jsonData.sum).to.eql(4);
});

One of the remarkable features is code snippets. This little test was created with them.

A couple of clicks + a couple of edits = the test is ready. Very novice-friendly!

As you can see, the results are colored and easy to understand.

Pytest

In pytest, everything you need is a test function and a standard Python assert operator (demo app).

import requests
from requests import codes


def test_two_plus_two():
    result = requests.post("http://127.0.0.1:8000/add", data={"a": 2, "b": 2})
    assert result.status_code == codes.OK
    assert result.json()["sum"] == 4

Test results are comprehensive right out of the box, even without tweaking.

So far, so good. Easy tests, quick solutions. Postman looks better as a graphical tool, and I love this “Test Results” section. On the other hand, pytest code is cleaner than Postman’s JS code. Postman test results are visually appealing, while pytest results are more detailed. No clear winner so far.

But wait…

Aren’t we comparing incomparable?

We need to stop and think about the correctness of such a comparison. Aren’t we comparing completely different categories, like apples and noodles?

On the one hand, yes. Postman is an API client/API platform, while pytest is a testing framework. There are a lot of use cases Postman can handle and pytest can’t, and vice versa.

On the other hand, both tools can be used (and they are used) for automated API testing.

And I believe it’s still worth comparing these “apples” and “noodles” because you, as a testing engineer, will have to stick to one primary solution if auto-tests become serious.

The cornerstone

As I can see it, the sources of the most problems with Postman autotests are

Postman, first of all, is an API client
The basic entity in Postman is a request

Auto-tests are just an add-on to Postman requests, not the primary thing.

While it might seem trivial, this approach is quite challenging for covering business flows and supporting auto-tests. You will see how things can get complicated further.

At the same time, full-fledged frameworks (pytest in our case) put tests at the forefront. A test is a basement; a request can happen inside a test or setup procedure.

Let’s dive into Postman’s testing approach.

Separated scripts

There are a bunch of scripts you can run before and after request execution. Different levels of the Postman workspace hierarchy define these scripts.

These scripts are stored in separate places, which makes working with code much harder.

In the interface, six sub-tabs are distributed among three tabs. These sub-tabs are not connected in the UI directly. So, when you click the “Send” button, you cannot say what happens in this hierarchy. You have to search among the tabs manually. And things can get even more complicated if we use inherited folders…

…or try to build a test script out of multiple requests.

What was happening to this variable?

Consider the basic CRUD.

Suppose we have a variable “new_world_name” derived from collection-level storage. In that case, we cannot say when this variable was created and what happened on previous steps without clicking previous steps and looking into each pre/post-script. And what if we want to rename this variable?

But how about Python modules?

“Folders and test scripts? But Python is the same!” some of my readers may say. Well, not exactly. A pytest module contains tests, not requests, with scripts glued together. You can easily see what’s going on in the test without jumping from one request to another. Suppose your code is distributed among multiple modules. In that case, you can easily find out how a variable is created and what’s happening in the test setup — thanks to prominent imports and pytest fixtures. Variables renaming and code refactoring, in general, is much easier.

Variable scopes and levels

A variable in Postman can be defined on multiple levels.

This scope model matches the spirit of Postman (remember levels of scripts?) but is fraught with several inconveniences.

Local variables

Let’s say we have the following pre-request script…

var company = 'Exness';

…and the following post-script.

var message = 'Hello, ' + company;
console.log(message);

Is it justified to see Hello, Exness in the Postman console? No!

The problem is that each script is run in a separate sandbox, so the local variables of one are unavailable for another. To share something defined in a pre-request script, save a value to a higher-level scope. Say, collection variables.

Pre-request

var company = 'Exness';
pm.collectionVariables.set("company", company); // saving to collection variables

Test script

var company = pm.collectionVariables.get("company")
var message = 'Hello, ' + company;
console.log(message);

Result

Pretty inconvenient, isn’t it? We will see how it can complicate things further.

You can try this by yourself: collection.

Of course, there is no such issue in pytest. Neither in body…

def test_variable_sharing():
    company = "Exness"  # define
    requests.get(...)
    logging.log(f"Hello, {company}")  # reuse

…nor in fixtures.

@pytest.fixture
def company():
    company = "Exness"  # define
    yield company  # Pass to a test
    logging.log(f"Hello, {company}")  # Teardown

Global variables work perfectly as well.

company = "Exness"  # define

def test_1():
    assert company == "Exness"  # 1st usage

def test_2():
    assert len(company) == 6  # 2nd usage

All variables are string ones

One annoying thing: in Postman, all variables on levels higher than local are strings.

// saving collection variables as local ones

var str_text = pm.collectionVariables.get("str");
var num_text = pm.collectionVariables.get("num");
var list_text = pm.collectionVariables.get("list");
var json_text = pm.collectionVariables.get("json");

// getting type of the variables and logging them

console.log(typeof str_text);
console.log(typeof num_text);
console.log(typeof list_text);
console.log(typeof json_text);

I understand why: the primary use case for variables is to be inserted into URLs and headers. But if you want to build a framework to ease routine, you can face a tedious process of repeatedly converting and parsing these strings.

// saving collection variables as local ones

var list_text = pm.collectionVariables.get("list");
console.log(typeof list_text);

// parsing to get object

var true_list = JSON.parse(list_text);
console.log(typeof true_list);
console.log(typeof true_list[0]);
console.log(true_list);

The demo collection is here.

Debugging

This is how we are supposed to debug our Postman code.

console.log(aaa);
console.warn(bbb);
console.error(ccc);

Remember the red breakpoint dot you used to use for pausing test execution in your favorite IDE?

Forget about it, Postman developers officially want you to debug via logging your variables.

To its credit, Postman console logs are impressive.

But they still cannot replace old good breakpoints.

If you hope to see this feature in the future, I’m here to disappoint you. The sandbox that Postman uses disallows implementing this feature. Period.

3rd party IDEs and code editors

Can we use some 3rd party solutions to mimic the IDE experience?

Collections export

The straightforward way to solve refactoring problems is to export a collection.

After that, you can use any text editor you like. Of course, it’s an inconvenient way to work with code.

It’s hard to read and easy to break.
There is no “run” button as well.
You must import it back and hope for the best after making changes.
You have to do these export/import actions each time

Postman Local

Postman Local is an unofficial tool that allows you to manipulate collections differently. It downloads and uploads collections via Postman API and stores requests and scripts locally per folder, making work much more manageable.

Pros:

The code is cleaner and intuitively separated, much better than raw exported files.
DRY. You can call JS code from anywhere. The utility will automatically add it to requests. Of course, it’s not your regular JS imports, but it’s better than the boilerplate you usually spread across collections.

Cons:

No breakpoints.
No run button. You must still convert these separate files into a collection to run afterward.
Postman environment variables are not natively integrated.
The project is slowly developing.

When I see this, I’m afraid to be stuck with an abandoned dependency in the near future.

I want to avoid going deeply into details. There is a comprehensive video I recommend to you in case you are interested.

I recommend trying it if you’re a die-hard fan of Postman. As for me, it cannot supersede pytest.

Modules and imports

What is the thing that (almost) any programming language has? Imports. Imports allow us to reuse old code while keeping it clean. You can always track where this class is from and how that variable has been created. Do we have something similar in Postman? No.

Postman VS Code extension

When I first saw the announcement, I thought, “Wow, this must be a game-changer!” Unfortunately, it isn’t. They implemented a cropped Postman interface inside the VS code for some reason.

What’s the point of having the same tool? It cannot ease the work at all. It can even worsen it, as many features have been left out. I hope they will allow us to use more IDEish things in the future. As for now, Postman Local, made by enthusiasts, looks much more helpful.

Conclusion

Both tools are suitable for the simplest auto-testing. But when you start to code in Postman, you face some struggles. One of them is variables. You cannot share a local variable between two scripts. All variables on levels above local are strings. This forces you to go through save-load and load-convert cycles.

After working with code in Postman, you may start feeling constrained, as Postman is not as good as other IDEs. Several solutions could help, but they can hardly get you close to a common Python+IDE experience.

In the next part of the series, we will explore how setup/teardown procedures can be implemented in these tools.