Feature Branch or Main? Exploring the Pros and Cons of Where to Test
Okay, you might wonder how in the world this photo below correlates with the title of my think piece. Bear with me; soon, you will find out all the meaning behind it. For now, think about this photo as an intro to the live-action movie with a transition to a cartoon representation.
Imagine, you as a QA Engineer have a new task/feature (large or small, complex or simple) on the table. How to decide on which branch to test it? Would it be on a separate branch or on the main branch after new feature merge?
The main branch, also known as master or develop, serves as the central location where all features are merged to form the entire product. For the purpose of this article, I’m going to stick with the name “main”.
Also, it’s important to know how many people will be involved into testing and, most importantly, how the decision you make will impact the product quality and your team’s productivity.
To determine the most suitable approach, consider asking yourself a series of questions.
- Will the feature be available for testing once the entire functionality is implemented, or can you split it into smaller sub-features?
- Is there a plan to release sub-features individually, or are the sub-features interdependent and intended to function only after integration into one comprehensive feature?
- How much time the QA team has for testing before the release date?
- Have you and your teammates included to the estimate the time for testing, stabilization (bugfix), and final verification of the feature/sub-feature?
With answers in hand, we can plan QA verification, detailing where it will be executed in terms of branches.
For the sake of simplicity, I have omitted detailed discussions about testing, bugfix verification, regression, and other pre-production activities that might occur in a real-life project.
In this article, I want to focus more on exploring variations that testers may encounter concerning branches, feature sizes, and feature complexities. Now, let’s delve into the three distinct blocks.
- block 1 is dedicated to testing on a separate branch(es): benefits and drawbacks;
- block 2 is dedicated to testing on the main branch: benefits and drawbacks;
- block 3 is dedicated to testing on a local machine (no branches as such).
Block 1: testing on a separate branch(es)
Here is my take on the pros and cons of testing on a separate branch or multiple branches from one or more developers.
Pros
- Stability and isolation of a separate branch.
New functionality won’t break stability of a main branch. Doesn’t not require feature flags.
2. Team capacity-friendly testing.
Testing on a separate branch leaves more space for maneuver especially when verification lies on shoulders of one tester and not a group of testers.
From the Pic.1 below you can see that the release happens once a day. However, the estimated time to finish the task for a new feature/sub-feature is 5 days of testing. If a tester verifies a feature on a separate branch, it doesn’t block or contradict the release schedule. Once the testing is complete, it can be merged and released on the fifth release day.
A group of testers will probably finish the testing earlier. Meaning, they provide a feedback faster and the stabilization phase with bug fixing can be done is a short terms as well. That depends on how many bugs were found by group of tester versus by one tester and which bugs were selected for fixing before the release.
3. Predictability and managability.
This approach enhances control over new feature stabilization (bug fixes, ensuring the feature meets requirements), minimizing main branch disruption from recent code changes; even if the task exceeds estimates, testing on a separate branch is more safe and tester ensures after verification it’s stable enough for merge and release.
In the example below (Pic. 2), you can see that a tester has a task with an ETA of 5 days. Verification cannot be done by the date of the first release, so it can be properly tested on a separate branch and included in the next release.
4. Early tester’s feedback to development team. If sub-tasks enter testing incrementally then it’s better to test on a separate branch and provide feedback promptly. Later on, tester can verify only bugfixes and integration with other new features or existing functionality on the main branch (after merge).
Cons
- Extended testing duration.
If a bulky piece of functionality was brought up into testing at once due to a specificity of a project and its implementation then it might be more time consuming if to test the same functionality in the feature branch at first and then re-test in main branch after the feature branch was merged. Due to lack of time, it’s makes more sence to test such a feature in the main branch hidden under feature flag. It’s especially true when only one tester is handling the feature verification.
2. Integration problems.
Conflicts between two separate branches. They can work fine individually but might not work when combined together.
3. Lost bugfixes after merge.
Due to errors when resolving merge conflicts there might be potential problems with bug fixes being lost and not merged into main branch.
From my practice, there were times when an already verified bug fix was not included in the main branch. When a tester is closing the ticket, they expect/assume that this fix will be included in the main by default, unless it was intentional, and there was an agreement to include the fix in another version of the product. In all other cases, a tester is not mentally prepared to discover a previously fixed and verified bug in the main branch, simply because it was not merged for some reason. It brings uncertainty and forces a tester to suspect that other bug fixes might have the same destiny.
In that case, you can check if a commit was merged into the main branch. If you cannot find the bug fix by the commit, then it means that the commit wasn’t included in the merge. When can the commit be changed for the bug fix? It can happen when several commits were combined (using the terminology of git “squashed”) into one commit. This way, the bug fix remains in the new commit, and there is no chance that the bug fix is not merged into the main branch.
Saying that, if you don’t see the fix on the main branch and cannot find it by the initial commit (as on a separate branch) in the main too, then it’s a good idea to go and ask a developer so they could assist and fix the problem.
4. Brand new bugs on merge
Picture this: a tester meticulously verifies a separate feature branch over several weeks, collaborating with the developer to fix every bug identified. After an additional round of testing, the tester gives the green light, indicating that the developer can merge it into the main branch. As a result, the developer may need to fix conflicts when integrating the feature branch with the latest version of the main branch.
Following this, entirely new bugs could surface within the new feature. A tester who was initially confident in the feature’s quality on the separate branch, may not be aware of newly introduced bugs after the merge unless there was a subsequent round of testing after merging the new feature into the main branch.
Block 2: testing on the main branch
Now, as we already went through pros and cons of testing on a separate branch, let’s see how testing on the main branch differs from a previous approach.
Be cautious testing something for the first time on the main branch before release.
In the given scenario (Pic. 3), a tester has a 2-day ETA task, and the next release is in 4 days, allowing time for completion. However, challenges arise in real-life scenarios due to varying estimates, feature complexity, and release schedules, affecting task estimation and testing completion.
Incomplete tasks may force rollbacks unless feature flags or separate branch testing is used. In case of not having feature flags the testing on the main branch can be distributed among a larger testing group for simultaneous completion. Another reason for rollback could be a feature not release-ready, insufficient time for bug fixes, feature stabilization, or it significantly disrupts existing functionality.
Based on my knowledge, here are pros and cons of conducting testing on the main branch (after merging a separate branch into the main branch), both with and without the use of feature flags.
Pros
1. Feature flags power and testing flexibility.
If a feature is hidden under a feature flag and merged into the main branch, there are less unpleasant surprises for a tester comparing to testing on a separate branch and merging it.
Feature flags help developers to isolate some modules/features/services from the existing functionality. It can be done by setting a conditional statement in the code with feature flag name. This is the way how the team can control the visibility of a specific features based on the state of one or multiple feature flags (enabled/disabled).
Everything that could be found during testing is a result of the interaction between existing and new functionality. There are fewer obstacles with testing since the feature is under the feature flag. The feature will remain invisible for the end user as long as needed up until the sign-off.
As an exception, there is a chance that existing functionality will be broken if the new feature was hidden under the feature flag incorrectly or partially. This can affect the release process in the form of regression bugs.
From the example below (Pic. 4), you can observe a scenario in which a feature, requiring 5 days of testing and stabilization, is tested under a feature flag in the main branch and can be released on the 5th release day. Even if the estimates were not accurate and the task took longer than 5 days, it doesn’t drastically affect or block the release process. The feature can be released later after it’s tested and ready for release.
If the feature is large and complex, testing often requires a significant amount of time and effort from the team. In this example (Pic. 5), releases occur every week. As depicted below, this represents a two-month testing phase for a feature hidden under a feature flag, which can be conveniently tested in the main branch before becoming stable enough for release.
On one hand, it’s beneficial to merge and hide a feature under the feature flag because it doesn’t require resolving merge conflicts like it happens in case with separate branches.
On the other hand, time does not stand still. After your feature has been merged and hidden under a feature flag, the code constantly merges into the main branch by another developer or team. This means it can potentially break some part of the feature once you enable the feature flag during your long-term testing. These issues arise at the intersection of two features where the code conflicts.
2. More confidence bug fixes are included.
Chances that all bug fixes will be included in the main branch after testing on the main branch are higher than merging a feature branch with bug fixes into the main.
Cons
1. Feedback delay.
It might take too long for the changes to be merged into the main branch. Consequently, we are losing valuable time during the changes’ PR review period, even though we could start testing earlier on a separate feature branch and provide early feedback, report bugs, and uncover unrevealed requirements.
2. Feature compatibility and integration risks.
Potential issues with a new feature may arise due to incompatibility with existing functionality, leading to significant problems in the product. If the release is getting close, there might not be sufficient time to conduct regression tests, address all bugs, and ensure thorough verification. The combination of these factors could potentially lead to a delay in the release process.
3. Potential risks to break main branch after merge.
The main branch with existing functionality can be completely or partially broken after merging changes from a separate branch. Depending on the processes in a project/company, there might be (or might not be) automated tests running on certain actions (merge, deploy, etc.). By having these automated tests, we reduce the chances of breaking existing functionality with new code. It is a good practice to run tests locally even before pushing changes to a remote feature branch. This is another layer of verification to reduce surprises in the main branch. Another way to mitigate the risks is to hide the new feature under a feature flag. While it does not provide a 100% guarantee, it reduces the chances of breaking the existing functionality on the main branch.
In this case below, you might notice that there are 4 days left before the first release day (day 4 on Pic. 6). However, the task to finish testing takes 5 days. Obviously, the feature cannot be included in the release candidate and should be included in the next (second) release. To achieve this, the feature should be hidden under a feature flag if the testing will be conducted in the main branch (meaning the untested feature was merged into the main). A similar case is described for Pic. 2 (when a tester conducts verification on a separate branch). As you might notice, you have a couple of options to choose from.
For comparison, check another example. Imagine we have the same release schedule and the same 5-day task to test the feature. But this time, the feature is not hidden under a feature flag but was merged into the main branch without prior testing. In this case, we either need to involve more testers and split the work if possible to meet the first release date with a stable feature (check the Pic.7), or test the feature on a separate branch before merging unstable code into the main branch.
There is a third option: include a not fully tested feature in the first release train. This option is the least acceptable but a very common case in the real world. Often, features could be tested but not fully polished and bug-free. Hence, some of them go into production for an uncertain period of time. It’s a completely different topic to discuss which defects’ priority can be deliberately omitted in production.
Block 3: testing locally
If you’ve already read so far, wow! I’m amazed! The first and second blocks are the main approaches that should be taken into consideration when choosing where to conduct your testing. But there is a third block that we need to mention here. And it’s about testing locally on your machine.
Unlike previous two approaches, testing the changes locally can be even riskier. Locally means you pull changes from the branch and do your testing without waiting for deployment on the staging environment. At the end of the day, you might not find some issues, as the local environment and staging environments are not the same, and a certain bug might occur only on staging.
Hence, such a bug might either be found later in the development lifecycle and potentially delay the release or even sneak into production, which is not ideal. The only benefit that lies on the surface is having a fast way to take a glance at a feature, not for the sake of full-scale testing but more for exploration or a small demo for other folks to whom it might be useful even in a raw state.
Conclusion
The choice of approach depends entirely on factors such as feature size (PR size), release schedule, and available testing time and resources within the QA team. It’s essential to adapt your approach based on the unique circumstances surrounding each situation. Be flexible and choose the most suitable tool for the specific case at hand. If you find this article helpful or simply enjoyed my drawings, feel free to show your appreciation with a clap. Obviously, those pictures are not generated by AI tool. :)
