API Testing Showdown. Postman vs Pytest. Part 3
In this part, we will try visually appealing Postman Flows, reusing code by storing it in Postman variables and exploring templates and built-in modules.
The previous parts of this comparison are here: part 1 and part 2.
Postman Flows
Postman Flows is a visual programming language that allows you to assemble requests into flows.
Let’s be precise: Postman Flows is not a testing tool, as stated by the authors.
Postman Flows is a powerful, low-code solution that empowers business professionals, product managers, and DevOps professionals to easily create, manage, and collaborate on API workflows.
But it still needs to be investigated as a potential solution for issues with Postman auto-testing.
I tried a simple CRUD flow to get acquainted with the tool. You’ve seen this collection before.
I don’t like that we have the exact GET requests for checking the results on each step. We already saw this: to avoid complicating things with setNextRequest, you must copy-paste requests repeatedly. I want to leave a single GET and try orchestrating execution with flows.
The first step is a world creation.
I ran it and saw a pleasant detail: my javascript tests run together with requests.
In the screenshot above, you see a routine status code verification. If you want to compare the initial POST /create_world response against the following GET to ensure the instance of the world got to a DB…
…it is hard to do this the “old” javascript way. This is because Postman Flows:
- Cannot pass data from a request to a variable available for a test script. This data can only be redirected to another request.
- There is a “Create Variable” block, but it is for sharing data inside flow only.
Fortunately, it is not the only option for testing; as I see it, it should be done differently. Behold the evaluate block.
This is self-explanatory. We have two values from different blocks, and we check their equality. The only non-obvious thing is the code field. Let’s say “Hi!” to Mr. FQL or Flows Query Language: a minimalistic and easy-to-start language. And it is even easier with this fresh feature — PostBot — a bot that creates FQL code based on your natural language requests. In particular, the code in my example was generated by PostBot. Possibly, FQL is too simple, and one day, you can realize that it is better to use JavaScript here.
Another fly in the ointment: if you deal with many tests and try to check a lot of stuff, your flow can quickly get cluttered.
How can you get the result of this verification? Flows don’t allow you to compile test results into reports. Another nail into the coffin of Postman Flows as a testing automation tool. Graphs can serve as a workaround:
Let’s try dealing with the problem I’m constantly talking about — boilerplate code. I added a PUT request and attempted to retrigger the old GET block to reuse all the verification I wrote earlier. It doesn’t work. The “signal” just cannot get to the GET.
Can we reuse the evaluation block? No. The considered result is the earliest one.
Don’t get me wrong, I understand that infinite loops and masked results are worse than that. I was trying the first most obvious hypotheses about doing this. And these dubious decisions are not required. For block sharing, we have groups.
If you duplicate a block, everything inside appears without interconnections or text comments.
And it is just a copy living its own life. I’d prefer to have an opportunity to create my block compiled of Flows’s primitives—something like functions in Python. If you must change in such a DIY verification or setup block, the change immediately spreads across all places. So, no DRY here. We don’t need to copy requests, but we have to copy blocks utilizing these requests.
The final collection is here. The flow is here.
Flows are unsuitable for testing, at least for now. Judge for yourself:
- No test runner. You must open each flow separately and click the “Run” button.
- No Newman support. You cannot run Flows from a CI/CD system. They are working on it, though.
- No test reports, only graphs.
- Working with variables is painful. A variable you took from is inaccessible from the JS code of a request’s test.
- No “functions” or custom test blocks, just groups that slightly ease copying stuff.
- Verification using FQL is a controversial thing. It is hard to imagine complicated verification with such a functional approach.
- “Evaluate” blocks themselves are also controversial. In case of more or less complicated responses, you may have to put a few just to avoid a mess in the FQL field.
- Money. Flows is only free to try on the Free plan. To use it in production, subscribe to the Professional tier. Pytest is free.
- The problem with almost any visual programming language is that it is much faster to type text than click-and-drag blocks.
Regarding the last point, the CRUD test on pytest can be much more concise. Let’s define fresh world creation as a fixture:
@pytest.fixture
def new_world():
"""Creates a new and empty world and returns its params"""
world_name = names.get_first_name()
add_world_response = requests.post(add_world_endpoint, json={"name": world_name})
return add_world_response.json()["world"][0]Also, let’s prepare a checker to reuse across tests
def check_world_state(id: int, world: dict | None):
"""Helper, GETs the world info and verifies its status"""
get_response = requests.get(world_endpoint.format(id=id))
assert get_response.ok
if world:
assert get_response.json()["world"][0] == world
else:
assert get_response.json()["world"] == []Now we can create short tests with no boilerplate
def test_new_world(new_world):
check_world_state(id=new_world["id"], world=new_world)
def test_put(new_world):
new_world_name = names.get_first_name()
response = requests.put(world_endpoint.format(id=new_world["id"]), json={"name": new_world_name})
assert response.ok
check_world_state(id=new_world["id"], world=response.json()["world"])
def test_delete(new_world):
response = requests.delete(world_endpoint.format(id=new_world["id"]))
assert response.status_code == 204
check_world_state(id=new_world["id"], world=None)The full code can be found here.
To be precise, Postman Flows is in beta so the situation can change in the future. One day, it will be a real salvation for those stuck to Postman. For now, Postman Flows is too limited for serious testing.
Anyway, if you want to learn more about Postman Flows, I recommend the official playlist on the Postman YouTube channel. That’s a good point to start from.
Code in variables
We’ve already touched on problems of boilerplate code you have to spread across Postman scripts.
The ways I see you can get DRY:
- Postman API
- pre-/test- scripts
- setNextRequest
- Postman Flows (to some extent)
- code in variables.
We’ve already covered the first four options. Let’s move on to the fifth one. As you have a full-featured JS VM, you can execute code from text variables using eval() or import().
As a practical example, we can get the BDD micro-framework from “Writing a behaviour driven API testing environment within Postman” by Shamasis Bhattacharya. By the way, I’m not a BDD guy; this is just a well-worked example of using JavaScript code from variables.
Let’s jump straight into the test function. The micro-framework is stored in a global variable and can be “imported” via standard eval.
eval(postman.getGlobalVariable("describe-it-test-script"));An example of usage of imported code is above. “describe”, “it”, “expect”, and “.toBe” are not the things available in Postman from scratch
describe ("response", function () {
it ("must have html content type header", function () {
expect(responseHeaders['Content-Type'])
.toBe('text/html; charset=utf-8');
});
it ("must return a 200 status code", function () {
expect(responseCode.code).toEql(200);
});
});So if you have some code to share across multiple test scripts, eval is a feasible solution.
What’s the problem with that?
- Unfair imports
2. Hard to edit and update.
- You cannot easily edit your code stored in a variable in Postman.
- You must do it by copy-pasting to your code editor or using Postman API / collection export-import. All options are controversial, especially compared to Python code editing in the IDE.
- If you use a 3rd party library, you cannot just npm update. You have to rip out the code from the library, insert it into your global variable, and hope it will work as expected.
3. As a consequence, debugging becomes even more challenging than in pre-/test- scripts.
4. Transitive dependencies are a pain. Suppose you insert a code of a 3rd party library that depends on other libraries. In that case, you’ll have to manually put these dependencies into variables and modify your desired library to “import” them via eval.
5. There may be some issues with loading large libraries. Remember, variables are for variables, not code; thus, no one guarantees that your library can be stored and used without friction.
Templates in Postman
One feature that can make your life with Postman slightly more pleasant is templates. When you create a new collection, you can choose a template.
My recommendation is the conditional workflow.
It’s another example of setNextRequest usage. Postman developers suggest you automatically switch between requests depending on the status code. Clean and minimalistic.
Don’t get me wrong, all postman’s problems are still relevant, and these templates cannot resolve them. But they can show you how Postman developers see their product used in practice. They also can be a starting point for a new collection.
Unfortunately, you cannot craft your templates. So templates, for now, are more educational tools than working horses. One day, they’ll allow us to save collections as templates, making complicated workflows much more manageable.
Built-in modules
You’re not obliged to use workarounds for importing 3rd party libraries, as Postman already contains some of them.
The following NodeJS modules are also available to use in the sandbox:
As you can see, there are a lot of verificators and parsers — the instruments required for testing. Unfortunately, the list is short and is not expanding often.
Conclusion
Unfortunately, none of the features we considered today can fix Postman’s problems with code sharing and extensibility. Postman Flows looks beautiful but can hardly be helpful for serious testing. Code sharing via variables may sound attractive, but several difficulties limit this approach. Templates in Postman are just another helper for novices and unsuitable for code sharing. Built-in modules are a pleasant detail but cannot change the game.
In the 4th part, I want to speculate a bit about DB<->REST verification and try some fantastic pytest features lacking in Postman.
