Democratization of Test Automation — An ING Story
A revolution in the making…
A stable and defect-free core banking system is paramount to the overall health of the economy. It serves as the backbone of financial transactions facilitating the flow of funds, investments and credit. A robust core banking system inspires confidence among consumers and businesses, fostering economic growth and reliability which we as ING Bank, take a lot of pride in. At ING we are committed to ensure that our banking systems are running without defects and operational failures, so that our customers get the best-in class experience, 24x7, 365 days a year, year after year! To achieve this, we focus a lot on the principles of quality assurance including Automated Testing.
This article delves into the transformative journey of empowering diverse teams with open-source tools and knowledge to seamlessly integrate test automation into their workflows. From breaking down technical barriers to fostering collaboration, this article explores the multifaceted impact of democratization of test automation. We will see how efficiency, reliability and innovation work hand-in-hand within the realm of software testing in ING. So read along with me as we unravel the dynamics of this paradigm shifting revolution, where quality assurance meets excellence.
The year was 2019 when it all began…
A Global Account Management System (in ING’s world this is synonymous to Core Banking) was undergoing an upgrade of 9 major versions. Normally in a banking industry, an upgrade of 2 or 3 versions in itself, is deemed quite humungous. But here we were challenged with 9 versions. In our “impact analysis” we had identified ~60 functional modules hence ~60 APIs along with ~300 Web GUI screens, but what was staggeringly imposing was the timeframe by which we needed to deliver this project as a team — 9 months!
2019.. 18th October .. 15:35 CET — That eventful moment when I was notified via an email that I would be in charge of the test automation wing of this project.
Working with me in the team were 2 brilliant “test automation” engineers — super enthusiastic, super energetic.. I knew I would be able to deliver this banking project with their contributions, but there was just one tiny problem — they knew nothing about banking !! They could brilliantly use effective design patterns to create super sleek test codes, but when it came to translating (for example) a SEPA Payment requirement into a testcase, I knew I needed to look beyond them.
If we take a closer look at the testing task on hand, we will notice that the Account Management System sits right in the middle of critical banking processes. So its not just about those 300 screens or 60 APIs anymore.. This upgrade was impacting many upstream and downstream systems, rendering the impact way more massive!
One thing was clear, this monster could not have been handled using age-old standard testing conventions and rules. After quite some head-scratching, we came up with 5 Fundamentals which would drive the automation story of this project. Little did we know, that these 5 Fundamentals were not just going to shape this project, but would, for the next 4 years, act as the foundation base for an entire gamut of projects in ING.
“The art of simplicity is a puzzle of complexity..” — Douglas Horton
- Easy and Simple — Straight out of the Agile Manifesto. The key to any successful project is how simple and easy can we make it for the people involved. In our case, we did this by applying the concept of “abstraction”. We created an interactive easy-to-use layer on top of our test code engine, which made it super simple for everyone to write good quality automated tests. I will talk about this more in details in the “Framework” section.
- Inclusive — It was imperative for the project to have all-hands-on-deck. So not just automation engineers but also Customer Journey Experts (aka. Business Analysts / Product Owners) who knew the functionalities inside-out, also started discussing different test case variations, environment constraints and even authored automated tests themselves. This was made possible with the help of an easy and simple framework as described in #1
- Early feedback with Early Testing — We aimed to shift left with our testing approach. As soon as a module was deployed we aimed to test it via automated scripts. This not only helped in providing a good test coverage (more test cases were getting executed faster and earlier) but also helped in improving the quality and stability of the automated tests.
Another important factor was our decision to test the APIs first instead of the GUI. Creating API tests are simpler and quicker. The tests themselves are quite stable and robust and run a lot faster than GUI tests. In our case this decision proved to be a game-changer. - Transparency — In order to remain transparent, and subsequently increase the velocity of the project, we asked ourselves 2 pertinent questions :
a) How can we keep the requirements of every module understandable by all?
The answer lied in creating data-driven tests with each data clearly explaining the requirement for that test case. In the example below the first highlight shows a testcase aimed to test a transaction 30 days in the past; the next one 30 days in the future; the next one is a negative scenario with mismatching debit and credit amounts and the last one shows a cross currency transaction.
b) How can we keep the test results accessible to all?
The answer lied in using Azure DevOps to its fullest potential. We used Azure Test Plans to store all test cases, so anyone could just open the test suite to understand what the coverage is and contribute by adding more test cases to it. We also used Azure DevOps Test Results viewer which displays test results (with Trends) and anyone can click deeper and deeper till they reach the failing test case. Here they can see in the “attachments” section the screenshot for GUI tests and request/response payloads for API tests.
5. Collaboration — This was the central piece of the puzzle. The entire project hinged on a super smooth collaboration model. We had members from Belgium, Netherlands and India — all committed to make this a grand success.
Here is a summary of the technical decisions we took, in line with the 5 Fundamentals we discussed above.
What followed was 3–4 months of rigorous cycle :
Design of Automated Test Cases → Execution via Azure DevOps → Enablement of Customer Journey Experts → Analysis of Reports → Feedback → [Repeat]
Hold on to this, for now. We will come back to this 2019 story in a short while. I am sure you would want to know whether we succeeded or not…
Before that, let’s take a look at a key factor which potentially changed the complete landscape of Testing in ING Wholesale Banking.
Enablement
When the goal is to ensure a good quality of the Application under Test, then it is imperative to have the testing and test automation, accessible and usable by all, specially the customer journey experts who with their knowledge about the application, can immensely improve the quality of the tests.
This gives the teams the ability to :
- Reduce their testing efforts
- Increase the overall efficiency
- Improve collaboration and communication
- Reduce cognitive load
Let’s take a close look at these 3 case-in-points.
#1 : Story from ING London—A team of DevOps engineers responsible for managing and maintaining ING’s wholesale business in the UK.
Business users and Chapter leads were quick to pick up the automation activities and started contributing more actively towards building regression testing pack. The quality related discussions with them became more goal oriented and there was a marked improvement in the automation process overall.
#2 : Story from ING Bratislava— A bunch of extremely knowledgeable individuals dedicated for testing connected systems, acting as the quality gatekeepers.
From manually testing lengthy end to end tests cutting across multiple systems, the team started using automation to reduce a lot of effort needed for manual testing and evidence gathering. They also began to look for more automation opportunities in high critical areas which were so far almost unheard of.
#3 : Story from ING Manila — A group of process-driven platform engineers responsible for operational activities like periodic deployments, monitoring and infrastructure maintenance.
With zero testing/test automation background, the Platform engineering team began to contribute to automated test creation and test execution for every Asian country rollout of a major product. This removed the need for dedicated test automation engineers from continuously being present in the team for maintenance purposes.
But how was it possible that so many engineers and functional analysts were able to pick up this art of automation with aplomb?
This brings us to the next protagonist of our story…
The Test Automation Framework
To understand the framework details lets go back to 2019..
From a testing perspective, The Application Under Test (The Global Account Management System) had quite a wholesome flavor:
- A Web interface through which various activities like system configurations and settings were performed.
- APIs, capable of performing major activities like payments, account opening etc.
- Database was getting upgraded with new schemas and tables. So this required connecting to the DB and performing various tests.
- The system generated loads of Functional Reports in the form of PDFs, CSVs, XLSXs and flat files.
- Message Queues like Tibco JMS and IBM MQ connecting the Application to multiple upstream and downstream systems.
So, we needed to build a framework which would cater to all these kinds of tests using opensource tools and yet make test authoring simple and easy by providing the end users the “abstraction” layer.
So this is how the framework architecture looked like :
The bottom layer consists of open source tools/libraries which are capable to tackle various technologies like :
- Selenium for Browser Testing. Playwright can also be integrated but back in 2019, Playwright did not exist yet 😉
- Java HttpURLConnection for API testing. This eliminates the need to import third party libraries like Rest-Assured
- JDBC drivers for testing various types of Databases
- Tibjms maven dependency (and other similar maven dependencies) for JMS testing
Over the last 4 years the usage of this framework has grown multifold. We have also seen a lot of demand to cater to various other types of testing. This calls for a constant evolution of this framework which we very lovingly call “INGenious”. We have added the following features to the framework also :
- Axe for Accessibility Testing
- JavaScript
window.performanceto capture the web page performance metrics, internal network API performances, etc. - Galen Framework for Web Page Layout Validation
- One-click integration to Azure DevOps to run tests via pipeline and use the AzDo Reporting features
As I am writing this article, Playwright is being integrated into INGenious and is being readied for an open-source global release.
Community
Before we see how the community around this framework and way of working has grown over the last 4 years, lets go back to the 2019 story and see how it really ended.
Tangible Results :
- Test cycle was reduced from 96 hours to 90 mins with a whopping 98% time saving
- Because of the ease of use, the automation was completed in ~4 months as opposed to the original estimate of 8 months
- We were able to go live with the upgraded version, 1 month earlier than the original planned date
Intangible Results :
- Collaboration : We achieved a humungous feat of bringing people across different geographic locations, time zones and professional backgrounds to work cohesively on this by creating a level playing ground for all.
- Process Improvement : Some concepts like “Shift Left” and “In-Sprint Automation” were put to actual practice here and the teams benefitted tremendously.
- Cloud Enabled : Azure DevOps was used effectively which made test orchestration and reporting very easy.
Cut to present day..
The community has grown organically. As I am writing this, we are now more than 100 engineers strong. The concepts and way of working has attracted attention not just from various tribes within ING Wholesale Banking but also from other business lines like ING Domestic Banking in Netherlands and beyond.
Teams have formed Testing Guilds and collectively take care of :
- Enhancement of the Framework
- Incorporation of feedbacks
- Documentation
- Propagate effective usage of the Framework, best practices in addition to sharing regular updates
By the end of 2020 we were the solitary team with these principles, ideologies and way of working, but by the end of 2023 we have 25 other teams joining the bandwagon.
As I stand here sharing this story with all of you, I am beginning to feel, its not just my story.. its not just the story of my team..
Its the story of all the individuals who kept getting added to this revolution over the years.. all those who have felt the power and autonomy to automate their regular mundane testing activities.. all those who could break the technical barriers and start writing better tests to improve the quality of the business product.
This story is a celebration of all those individuals and their achievements!
If you have made it this far, thank you for reading along.. This is Ashish Ghosh, signing off..
Happy Testing !!
