Introducing Functional Test Automation In Your Company
Introducing Functional Test Automation In Your Company
Most of the time starting something new in any sphere is quite a difficult thing to do. The same goes for introducing automated tests as many things have to be taken into account when choosing the framework and tools to be used for automation:
- needs of the company
- technical skills of people who will be writing the tests
- technical stack of the company
- details of projects for which test automation will be introduced
- requirements for test run statistics
- etc.
We will discuss these variables that have to be taken into account in more depth and have some comparisons in doing so. Also, our train of thought and the conclusion for choosing the programming language and frameworks for automated tests will be provided (without going into too many details).
Lets start off with company specifics
Different companies have different requirements and needs for test automation (obviously…). First off, evaluate what kind of projects your company is working with. Are they mobile applications? Maybe web applications? Or is it something more exotic like desktop or linux CLI apps? Depending on the answers to these questions you will already have an answer with a set of available frameworks for your test automation. There are some frameworks that allow you to write tests across all platforms (such as Appium), but most of them are either for web automation only or for mobile automation only.
Then you should ask yourself what kind of automation do you want? By saying what kind I mean record and replay or writing scripts for your auto tests. The answer to the latter question may depend on the competences of your QA team and their willingness to develop one or other type skills.
Evaluate your QA teams skills
One of key things that have to be discussed are competences of current QA team. To do this you should have answers to some questions.
- Are they familiar with test automation? If so to what extent?
- Are they capable of writing a test script by themselves?
- Are they capable of setting up test automation?
- What frameworks have they used?
- What supporting tools (for test runs, reporting, notifications, etc.) have they used? This might also be a key point to selecting a programming language to be used and the framework to be used.
Usually if there is some know-how of the team on using a specific framework, especially if they are fully capable in customizing it, you would not want to make a decision of choosing a framework the team is not used to without having specific benefits of the new framework that would outweigh the teams knowledge in another framework.
If there is no knowledge on any framework from the team then they key question that has to be answered is their willingness and motivation to learn either record and replay tools or write test scripts and configure all the necessary integrations (test runs, reporting, notifications, etc.).
Record and replay vs. test scripts
Generally, record and replay are tools that allow one to easily create auto tests by saving the actions that are performed by the user and saving them into an automated test. Also, most of the time they will have a set of functionality that will help you setup test runs, reporting, notifications, etc. One thing to have in mind when using record-and-replay tools is that using them requires more time spent on maintaining automated tests.
Writing test scripts is more difficult than using record and replay tools as it requires a bit of knowledge in one or another programming language and needs more input from the person developing automated tests as each step and each assertion has to be written in the test script. Frameworks used for writing test script usually do not have fixed functionality for setting up test runs, notifications, reporting and all these things have to be configured separately. On the other hand, they are easier to integrate to already existing environments/tools (e.g. notifications to slack, integration with CI/CD server and so on). Availability to select customized test reports, other tools to integrate your tests with is one of the biggest pros in using test scripts versus using record and replay tools. Since our company decided on writing test scripts it will be discussed in more depth.
Influence of technical stack used by developers
Is this really a relevant thing when discussing automated tests written by QA engineers? Does it make sense to somehow link selection of tools used for automated tests with what the developers are accustomed to using? These are probably questions that pop in your mind after reading the section title. Short answer to this question is yes, it makes sense. Especially when choosing to write test scripts. Let me elaborate on why this makes sense.
First of all, having developers who can actually give you advice on how to setup a framework, select libraries to use, set up CI/CD, set up other integrations is a great thing. At the very least you will have someone who might be at least distantly familiar with the tools that QAs are going to use for test automation and in some cases maybe even use themselves.
Another thing that can be leveraged is the programming language knowledge. This way you always have someone to go to for code review if there is only one QA initially writing automated tests.
Probably if one would think deeper on this topic, there would be even more pros in QAs using similar tools to those used by developers.
Influence of projects to be automated
Here obviously you have to consider the platform (or platforms) of the projects and frameworks on which they are written (for example mobile applications developed with flutter). Looking further on details of the project have to be taken into consideration as there are test automation frameworks that are not capable of testing certain functionalities. From our experience most of these limitations appear on JS based frameworks (Test Cafe, Cypress and maybe some others). One such example could be testing web applications that redirect to a different domain or open a link in a new tab. This usually causes challenges when writing tests with Cypress and in most cases you would have to intercept these requests to check if they are working correctly or use cy.origin(). Web pop-ups (alerts, confirmations and prompts) are also something that needs a bit more work to test (an example of dealing with them for Cypress can be found here). Another challenge is conditional testing on Cypress. In most cases it is impossible and in other cases it is quite difficult at least. These are some of the limitations that a framework can have from the functional testing perspective.
There are some other things to consider as well, such as test suite size, use of API calls and calls to databases from the test scripts, available integrations with different CI/CD platforms, reporting needs, collecting test run statistics, availability in setting up test parallelization, etc.
A real life example in choosing test automation tools
Here I will try and give a short summary of introducing test automation and some related tools to a company by providing some insights into the details that were discussed up to this point. Lets start with the competences of current QA specialists. Most of them have limited knowledge on test automation, but there is motivation to learn something new and develop their technical skills. So here comes the first decision (to some extent) — choosing to write test scripts instead of using a record and replay tool. This was also impacted by want to have more freedom in choosing the supporting tools for automated test.
Now lets talk about the projects. The company has diverse projects for mobile applications and web applications. In most cases the project is either a mobile application (usually on both Android and iOS) or a web application and only few cases have both mobile and web applications.
Talking about mobile application stack some of them are native applications and some are developed with flutter. Since there is quite an easy train of thought from this point for mobile applications I’ll discuss it in a bit more depth right now. First of all, mobile applications don’t have as many frameworks to choose from. Most commonly heard ones are Appium, Detox and native automation tools (Espresso and XCUITest). Now having in mind that a single QA specialist usually test both Android and iOS applications and the fact that some mobile applications are developed using flutter the choice here becomes quite clear — lets use Appium that allows writing automated tests for both Android and iOS applications and has a way to write automated tests for flutter application by using Appium Flutter Driver.
This answers one of the main questions — what framework to use for mobile test automation. But what about the programming language that will be used to write these tests? Appium gives us freedom to choose from basically any language we want. To answer this one lets jump to web automation.
Now this becomes a bit more interesting, as there are tons of different frameworks to choose from of which most are bound to a single programming language. Cypress, Test Cafe, Puppeteer, Playwright, Selenium, Appium, Robot framework, WebDriverIO and many more. How to choose from all of these options? What to take into account? Only the limitations of functionality that can be tested by using one or another framework? Maybe ease of customization of the framework or initially available functionality? What about efficiency and speed of the tests? Maybe stability and not having flaky tests is a key thing for you? These are some of the variables that should be taken into account at least to some extent.
The key variable for us in choosing web automation tool was ease of use. You might say that all of them are rather easy to use, but are they really? Some of them deal better with certain cases than others. An example could be getting stale element exceptions, for selenium this is something that may cause tests to be flaky and require extra effort to heal that flakiness. Dealing with page load times, scrolling to elements, waiting for certain elements to load on the page are to be considered. From what I have encountered personally is that JS based frameworks usually deal with these issues best. Considering this and the ease of use fact, a decision was made to select Cypress as the framework for web automation.
With choosing Cypress for web automation and wanting to have as much consistency in test automation as possible a decision to use JS (with Appium) for mobile automation was made. Using frameworks developed on JS gives quite a lot of freedom in choosing libraries that you would want to use for test reporting, customizing your test framework and implementing other tools to be used in test automation (such as device farms or email testing services).
