Automating Tests for a UI-less System
As a Test Engineer at Insider Inc., I work closely with the company’s Membership Identity team, which oversees our user and subscription databases and associated codebases. As part of my job, I’m responsible for writing automated tests to verify the functionality of the Membership Identity system. Here’s a brief rundown of how I helped integrate automated tests into our system.
Our system consists of a series of microservices that communicate through AWS Lambda functions, distributed across multiple codebases. When tasked with architecting an automation suite to test the integration of these services, I realized that the methodologies in play would be significantly different than the Selenium-based browser automation I had worked with in the past.
Before my work could truly begin, I had to answer several questions: Where would my code live? What language should the tests be written in? What would be the best way to track the flow of information through our systems? How could I remove third-party dependencies from the internal systems being tested? How could data be reset for each test run without affecting the “source of truth”? Could I integrate these tests into a CI pipeline to ensure tests would be run when new code is deployed?
First things first: where would test code live?
On other projects our test code resided in the repository being tested, but the new system had no fewer than three repositories at the time testing began, and the potential for more to be added as the project matured. For the sake of clarity and simplicity, I decided that the tests would have their own repo and not be a part of any of the individual codebases under test.
Which programming language to use?
Shortly before embarking on this project, the Test Engineering team decided to abandon Java in favor of JavaScript for our web application testing. As our microservices were predominantly written in Node.js, JavaScript appeared to be a good fit for this automation suite as well. Using a language familiar to both our team of Test Engineers and project developers opened up opportunities for collaboration and pair-programming we would not otherwise have had. This also follows a recent trend of companies switching to JavaScript frameworks for browser testing.
What are we looking for?
When testing these distributed systems that power our site’s handling of user and subscription data manually, we check a combination of API response codes, returned JSON objects, database entries, and emails indicating the successful transmission of Simple Notification Service messages. Amazon Simple Notification Service (Amazon SNS) is a messaging service for both application-to-application and application-to-person communication. SNS topics are push based and can be subscribed-to by both applications and individual users. While we did not attempt to integrate these email notifications into our automated tests, we were able to assert data points for every other piece of the system that we would check manually. A typical test progresses like this:
- Setup data for clean test run
- Send payload via the API
- Assert expected response status code
- Assert values on returned JSON object
- Assert changed database values
- Locate logs for internal system event in database
- Locate logs for third-party event in database
As the logs saved to the database are comprised of the same data sent in the SNS topic confirmation emails, we are essentially able to verify that these messages are transmitted without the email confirmation itself.
How to cut third-parties out of the loop?
Our microservices are not a closed system — they also communicate with third-parties that both produce and consume data that travels through our network of services through a series of webhooks. For testing purposes, we’re really not interested in interacting with these third-parties to trigger our scenarios, so we sought to cut out the middleman as it were, and post to our API with payloads mimicking what we would receive from our external partners.
However, we ran into an immediate roadblock: all webhook data from one of our partners was encrypted with a proprietary algorithm that we don’t have access to, and this was the only type of payload our API would recognize on the applicable endpoint. This meant that I could not simply generate a new payload for each test. Fortunately, the third-party in question retained a copy of the encrypted webhook payload for each transaction, so I was able to locate examples of each webhook type that we wanted to test, and set them as constants to be used as payloads via SuperTest (SuperTest is an HTTP assertions library that allows you to test Node.js HTTP servers). It is built on top of SuperAgent library, which is an HTTP client for Node.js. It is used in our tests to connect to REST APIs and assert against returned points of data like the status code.
Retaining the “source of truth”
When our system receives an encrypted webhook, it does not directly consume the data contained therein. Rather, it treats the webhook as a prompt to connect to the third-party and consume whatever data is present in their system for the associated account. If I have a webhook payload creating a new user test@test.com, but I change the email address to testing@test.com on the third-party’s site before sending the webhook, the user it creates in our system will have the email testing@test.com.
These changes also propagate both ways. If I use our API to update a user’s email address, it will get updated on our third-party automatically. For the purposes of our testing, since we are using static webhook payloads, we need to be sure that the third-party data remains unchanged for the accounts being tested. Therefore, we cannot use our API to reset user data in our databases prior to each test run, as that would also alter the data on the third-party’s end. Instead, we connect to the internal databases directly and use queries to locate, alter, and sometimes delete entire records as a precursor to sending the webhook payload through the API. This allows us to see changes to, or creation of, the same records over and over without affecting the original source of the data.
CI pipeline integration
Having a suite of tests is great, but if they don’t run regularly they won’t do much good. We already used Jenkins on this project to manage releases to our development and production environments. It then made sense to leverage the existing Jenkinsfiles in our non-test repositories and include an Integration Test step. This step checks out the test repo and runs the automation suite after code deployment when the environment being deployed to is development. One challenge of this effort was to mask the passwords to our databases in both GitHub and Jenkins, which was solved by the implementation of a .env file to store sensitive data. Now, every time new code from any of our repositories being tested is deployed to development, the test suite runs and gives instant feedback on potential issues, reporting test results to a dedicated notifications Slack channel where they are easily accessible and actionable.
Reflections and looking forward
On the whole, I feel like working on this automation suite was a real learning experience and definitely a departure from the “load, wait, assert, click, assert” patterns common in UI web testing that I was already familiar with. Testing a system with no front-end required a deep understanding of how our microservices interact, both with one another and our third-parties. I’m excited to see this suite grow as the Membership Identity team adds more services to our ecosystem to better serve our users, and I look forward to collaborating with my teammates on new automated testing challenges to come.
Looking for a new job opportunity? Become a part of our team! We are always looking for new Insiders.
