Smalltalk about trunk-based development
As a technical lead, I have regularly conducted job interviews with software engineers in various companies, and over time I have become accustomed to mentioning when trunk-based development is taking place in the engineering team. The reactions to this always make me smile inwardly, because they vary between amused, amazed, perplexed, curious or sometimes delighted. Not so seldom I am told why trunk-based development cannot work or is even dangerous and downright the devil’s work. I then allow myself the discreet remark that both smaller and large engineering teams have been using this method extremely successfully for years and that it is by no means something new or unproven.
In these conversations, however, I always notice a pattern, because very often only the method of trunk-based development is considered in isolation. And in this case, I have to agree with the critics: Pushing directly to the main branch, without a CI/CD environment, without automated tests and without a four-eyes principle can definitely be called hara-kiri.
This article is therefore not exclusively about trunk-based development, but about a technique that only works in conjunction with a number of basic requirements and presupposes a certain team structure and culture.
How does trunk-based development work?
The basic principle of trunk-based development is amazingly simple: instead of using feature or release branches, changes are brought together by the developers directly on the trunk (also called main or obsolete master). It is not uncommon to find teams that work trunk-based without a single branch, but develop solely on the main branch.
In such a setup, code changes are merged at short, regular intervals and usually deployed directly and automatically. This avoids extensive change histories and promotes a working methodology based on small, continuous developments with a high degree of automation.
The benefits of trunk-based development
No development method is a panacea and comes without advantages and disadvantages. Nevertheless, trunk-based development minimizes or prevents some of the typical challenges that are usually encountered with other development methods.
Fast release cycles
In companies that work with Git Flow or a variant of it, it is not uncommon to find release cycles of several weeks or months. However, this severely restricts rapid action that is geared to changing market conditions. Trunk-based development supports companies in the continuous, time-independent release of new functionalities for their product.
So if software is developed with an MVP (Minimum Viable Product) mindset, trunk-based development helps achieve these goals through frequent releases. It’s about understanding and satisfying user needs as quickly as possible while ensuring continuous development. New functionality can be made live immediately, and the high degree of automation provides additional support through reliably working tests and reproducible (and rollback-capable) deployment steps. High-performing teams working according to modern principles therefore often use trunk-based development to achieve fast progress while maintaining high quality and high learning success.
Frequent committing for stronger commitment
The high speed of development, pair programming, automated testing and associated approaches such as Test Driven Development, and ultimately the independent responsibility for deployment to production place high demands on software engineers. An extensive understanding of the various interlocking complex systems, such as CI/CD pipelines, infrastructure-as-code, automated test suites, container runtime environments, and more, is required to successfully assemble all the pieces of the puzzle into a functioning service. This, in turn, increases the commitment of the individual team members, because they see their direct impact on the (further) development of the product – in a positive as well as in a potentially negative sense – if, for example, bugs occur.
The silo thinking between Dev and Ops is broken down and the result is a DevOps mindset in which employees can and should take responsibility. This results in a deeper understanding of all people involved for the technical processes as well as the business processes, and communication barriers are reduced.
Continuous and synchronous code reviews
The constant integration of code changes inevitably requires a reliable review system. Continuous code reviews support high code quality and knowledge sharing between software engineers (which in turn helps to prevent knowledge silos). A four-eye principle also increases the speed of the reviews and reduces time delays that often occur in code reviews via pull requests when communication does not take place directly but via written comments.
Avoid the “merge conflict hell”
Hand on heart: who hasn’t cursed when tons of merge conflicts arose during the merging of two branches, which often have to be laboriously resolved manually? Trunk-based development puts an end to this meaningless but time-consuming work and puts the focus entirely on the simple integration of your own code changes without headaches.
The basics
It is essential to understand that trunk-based development alone is neither a panacea nor will it work reliably. The working method is part of a development style that is strongly inspired by Extreme Programming. The guiding principle is a reactive, fast development team that gets by without obstacles such as lengthy merges and a jungle of long-lived, outdated feature branches. The latter in particular often leads to countless different development statuses or, in the worst case, even unclear code versions on the different environments.
The CI/CD environment as the centerpiece
CI also stands for Corporate Identity in marketing language, but in the engineering context we speak of Continuous Integration. CD means Continuous Deployment and is the second step in a CI/CD environment. The basis for such an environment is CI/CD software such as the open source software Jenkins or SaaS solutions such as CircleCI or the CI/CD platform integrated in GitLab.
In the CI/CD environment, a process (often called build pipeline) is automatically started with each new Git commit, which normally takes over the following steps automatically:
- A
git cloneof the last code state is executed on the configured branch. In the case of trunk-based development, this is usually the main branch. - Depending on the programming language, a build artifact is created first (for compiled languages such as Java, C++ or Go). For runtime-interpreted programming languages such as Python, Ruby or PHP, a Docker image is usually created that contains the latest code as well as libraries and other dependencies and assets.
- The automated tests are executed in this Docker image. If a test fails here, the execution of the CI/CD pipeline is usually aborted, as no stable deployment can be guaranteed.
- The code changes are integrated one after the other into the different runtime environments (development, staging, production). For example, this may be the web server that executes the code for a web app.
- The integration may also include rollback-enabled database deployments if, for example, the database schema has been changed.
- In the case of an automated server infrastructure, the code for infrastructure provisioning and management is usually integrated as well, especially if an infrastructure-as-code approach is taken.
The runtime of all these steps should ideally be kept as short as possible so that the pipelines do not represent a bottleneck in the deployment process.
Fail Fast: automated tests
I already mentioned it at the beginning of this article: trunk-based development with direct deployment to the production environment would be hara-kiri. It is job of the CI/CD pipeline to make sure that the automated tests are executed on every commit. If a test fails, further execution of the pipeline is aborted and the build is in failed status.
Automated tests ensure that relevant components in the code are tested in a sustainable manner without getting lost in implementation details, thus driving the costs of maintenance up and the developers crazy. The different levels such as unit tests, integration tests or end-to-end tests are ideally executed in the CI/CD pipeline in the order corresponding to their costs in a time sense. This fail fast approach has the goal that we as developers get feedback as quickly as possible when a test has failed and the pipeline has been aborted. It makes no sense to run the expensive end-to-end tests first, only to fail a simple unit test after waiting for minutes.
As a developer and in the context of quality assurance, I want to know if my code changes could affect existing functionality. At the same time, agile development approaches primarily pursue the goal of reacting quickly and flexibly to changes and rolling out new functionality easily and securely. Manual tests (i.e., tests performed by people) are obviously unsuitable for this purpose, since they are inherently error-prone themselves on the one hand and there is no synchronicity on the other. Automated tests, on the other hand, are always executed stupendously by the CI/CD software according to the same pattern and – if implemented well – ensure that the tested workflow still functions as defined in the tests even after changes.
Safety through code reviews
In Git Flow, long-lived feature and release branches are part of the concept. Due to their sometimes difficult-to-understand complexity and interdependencies, they can represent a serious weak point in quality control. The smaller batches of code changes in a CI/CD-based approach, on the other hand, are much more realistic to understand during a review. The pair programming approach, also inspired by extreme programming, goes one step further here, with two developers working together continuously and a permanent dual control principle being guaranteed. In this way, errors can be detected before they even reach the codebase, and the resulting errors in development can be avoided.
Regardless of whether code review or pair programming is used, synchronization is also important here in order to ensure a fast, continuous development flow. Pair Programming is a bit ahead in this respect (besides some other advantages), because there is no interruption in the workflow. Two developers write code together, prevent errors in the codebase through the dual control principle, run the test suite locally, push their change and only have to make sure that the CI/CD pipeline is run error-free. The time required: a few minutes from the commit compared to often hours to days for the classic Git flow with pull requests and time-consuming code merges.
Container-based runtime environment
While it doesn’t necessarily have to be Docker (Podman or Buildah are also runtime environments that run as containers at the OS level), I’ll limit myself to this top dog here. Services that exist as container images can be rolled out to the production environment relatively easily as part of the CI/CD pipeline, for example, within a Kubernetes cluster, Amazon ECS or similar infrastructure. Container-based runtime environments are not a must-have, but they make it much easier to work with a CI/CD environment and thus indirectly support trunk-based development.
Infrastructure-as-Code
Seeing the infrastructure as part of the codebase is now best practice in the Ops world. Here, the management and provisioning of infrastructure resources such as servers, databases or DNS configurations is handled by tools such as Terraform or Pulumi. The configuration is written as declarative code and uses the APIs of cloud platforms such as Google Cloud or AWS to make incremental changes to the infrastructure.
As part of the CI/CD pipeline, these changes to the infrastructure are also rolled out automatically and are thus part of the entire ecosystem, which is developed trunk-based.
Modular software architecture
A modular application architecture is not a mandatory prerequisite, but it is certainly helpful. Whether microservice architecture, SOA architecture or monolithic architecture – trunk-based development basically works with any of these approaches. However, a modular architecture geared to smaller services clearly helps to minimize dependencies, reduce the runtimes for test suites and significantly simplify the deployment itself.
The human factor
The role of the people working in a team built around the CI/CD environment should not be underestimated. In particular, techniques such as trunk-based development, pair programming, TDD or the assumption of responsibility even beyond a pull request are ultimately procedures that must be actively lived every day. The impact of the team culture of such engineering teams on their own success can therefore be described as significant.
High-performing teams attract people who are willing to think outside the box. Thinking in silos does not work there; the willingness to assume responsibility is a basic prerequisite. Taking responsibility means, above all, that as a software engineer you are always aware of being responsible, together with the pair programming partner, for writing scalable and automated tests, well-understood code and the rollout of this code. Only when a release has been successfully completed after the steps just mentioned is the fundamental work done. If, on the other hand, the CI/CD pipeline fails, it is the developers’ job to ensure that the underlying problem is fixed as quickly as possible.
In summary, working in such teams is particularly appealing to people who can master the pragmatic balancing act between foresight and attention to detail, enjoy working across teams, and are able to communicate actively and collaborate frequently. A few of the team members should be experienced senior software engineers who can regularly collaborate with their less experienced colleagues and share the do’s and don’t’s around trunk-based development.
Best practices
Now that we have taken a look at the basics that are important for a functioning trunk-based development environment, we will focus on the equally relevant soft facts. Trunk-based development only works within an appropriate team culture with the right tools and by following a few rules.
Ensure high availability of the build pipeline
Trunk-based development is accompanied by continuous new deployments and therefore requires a highly available and functional pipeline for each service. Errors in the CI/CD pipeline can of course occur, but they should be corrected immediately by the respective developers. Likewise, the rule should apply that no new commits may be made to an already red build. It is therefore up to the developers to check the status of the pipeline before each new commit.
Use feature toggles
Feature toggles (or feature flags) are an important element when working directly on the main branch. In their simplest form, they are configuration variables that enable or disable certain functionality on the software side. Instead of using separate release branches, they can be used to exclude parts of the code from execution. They make it easy to activate certain functionalities, for example in the development or staging environment, while the production environment may not yet be aware of the new feature.
Deploy smaller change batches continuously
With a functioning CI/CD toolchain, Continuous Integration can be lived to the extreme and code can be deployed several times per hour. Of course, an adequate amount of automated tests must exist for every deployment. As a minimum, the rule of thumb should certainly be to deploy at least once a day, but ideally more frequently.
There is also a psychological effect behind this: every deployment automatically triggers a release into the production environment. Every software engineer should know this workflow in his sleep and thus be able to build up a high degree of trust in the infrastructure. The less often a deployment is made to the production environment, the more risk-averse the people involved become. If, on the other hand, this type of deployment is business as usual, the fear decreases and the trust increases, and with it the motivation to contribute to the further development of the software.
Write. Automated. Tests.
I suspect that the aversion to trunk-based development mentioned at the outset also stems from a certain respect for the complexity of the automated test suite required for it. There are unfortunately still far too many applications, written by too many developers, that don’t include any testing at all.
One thing is clear: getting to grips with the necessary techniques such as mocks or stubs, fixtures or factory patterns, as well as understanding the different types of tests in the test pyramid, requires a certain amount of time. And of course, it’s more interesting to keep banging out new features instead of dealing with the potential negative consequences of your code (comforting note: this is all too human behavior and is called cognitive dissonance). However, the inconvenient truth is: as a developer in a professional environment, it is part of our duty to also ensure that our code is scalable, maintainable, extensible, and robust. Used correctly, automated tests are the most powerful tool to achieve these non-functional requirements.
Keep your build pipeline clean and fast
Nothing is more tedious than having to wait an hour for a new commit before the result of the build pipeline is known in the CI/CD environment. Often, long build and compile processes or imperformant test suites are to blame for the misery. Try to keep the time from commit to successful deployment to production below 10-15 minutes and consistently invest time if these times lengthen. Build pipelines are like democracies: we need to keep an eye on them and maintain them, otherwise they slowly become a problem case and slow down further development.
Don’t use branches
If you manage to reduce the development setup in your company to one, two or three branches per repository, you can probably do without them. Trunk-based development means developing on the trunk and you should not soften this fact. Try the approach in a smaller project if you are unsure – but follow through with the setup in a way that fits the actual concept behind it.
A personal opinion on trunk-based development
There are probably tons of pro-contra comparisons, a lot of already formed opinions and partly also a certain emotionality in this topic. In the end, trunk-based development primarily helps to put the focus of a development team entirely on developing software. Engineering teams are understandably a black box for many non-techies, and a relatively expensive one at that. The widespread understanding reads as follows: There are these developers, they get requirements, turn them into code, test the code and go live with it.
A development methodology built around Continuous Integration and Continuous Deployment is relatively close to this basic understanding. It is hard to explain outside the tech world why complicated releases take hours or even days, what release branches or tags are, or why it takes several different roles (and thus mostly people) to complete the go-live of a small feature.
The simpler and more logical an already complex process is, the better. Trunk-based development supports this and comes relatively close to this ideal. Using standard technologies and following best practices are serious considerations in order not to fall into the builders’ trap, because even in the infrastructure area (and I include the CI/CD environment in this) one often encounters complex, self-developed setups that devour corresponding costs for further development and maintenance, but play only a subordinate role for the business case itself.
It is not uncommon for engineering teams to have a certain over-engineering tendency. This is not fundamentally reprehensible, but it does add up to a considerable amount of additional effort and the associated complexity in the tech stack. With the mindset required for trunk-based development, this behavior pattern can at least be somewhat prevented. Continuous code reviews, task sign-offs and the four-eyes principle define a kind of north star that constantly reminds each team member of the most important and relevant core tasks and goals.
Conclusion
Fortunately, it is not rocket science to build a development process based on commits to the main branch. However, a few basic requirements should be created and observed in order to make the associated benefits such as higher speed or more constant code reviews possible. The personal attitude of the members of the development team plays a role that should not be underestimated, because trunk-based development means one thing above all: wanting to take responsibility and building up a broad understanding of software development, in which the entire lifecycle of the code is a supporting component.
However, once you have become accustomed to this high-performance and, in most cases, uncomplicated way of working, you no longer want to do without it. The knowledge that you are not wasting your time on irrelevant and tedious merge processes is perhaps enough of an incentive for some people to look into modern forms of deployment.
