Key Takeaways
|
As CI/CD pipeline usage grows in organizations, complexity also grows. Through my work with organizations across industries, I have seen the full range of CI/CD maturity patterns in the enterprise. From low maturity pipelines to high maturity pipelines, I’ve found a new pattern that helps organizations reduce complexity once reusable templates have become too complex to maintain: Generated CI/CD pipelines. In this novel approach, CI/CD pipelines can be generated programmatically, yielding a predictable result, easing maintenance and ensuring highly reliable pipelines.
Technical leads are quietly figuring out the cost of maintaining their CI/CD pipelines is higher than expected. Developers are spending as much time on fixing CI/CD toil as they are writing new code and features. This has recently been corroborated by Chainguard, Harness and the IDC survey.
CI/CD pipelines automate build, test, validation and deployment workflows to support continuous integration and delivery. They help deliver code faster and more reliably. Those automated steps used to be run manually on the software developer’s local machine. DevOps methodology integrates them into the source control platform (your GitHub or GitLab instance), which solves the “it worked on my machine” syndrome. This kind of automation helps ensure build reproducibility, reduces the risk of manual error, maintains consistent quality between releases, and enables faster iteration and deployment.
Organizations usually start small, with a single developer or a single development team. As they grow, they do not always pause to deliberately think about how their teams, platforms, and delivery practices should scale. Instead, they solve problems as they appear, adding tools, workflows and exceptions one at a time. Without a clear operating model, the kind of thinking described in the book Team Topologies by Matthew Skelton and Manuel Pais, this incremental growth can gradually turn into organizational and technical complexity.
This leads to some blind spots. One frequent pattern we commonly see is duplicated CI/CD code per project, rather than reusing a template tailored for a specific technology stack. Duplicated code makes it hard to share improvements between projects and to implement new features in a uniform manner. It also increases the time required to deal with security issues, makes it more difficult to enforce compliance and longer to onboard new developers to a new project.
When we do see clients use templates (what is called reusable workflows on GitHub), they are often complex, which makes maintenance more difficult. Reuse solves many of the problems seen with the “unique workflow per project” approach, but when those templates have to support a large number of use cases, maintenance can become so difficult that expecting to modify them without breaking a project becomes trial and error.
On top of the previously mentioned side effects, velocity decreases. The flow when doing changes to a CI/CD pipeline is the following:
Waiting for a runner to pick up the last commit can take a few seconds. Waiting for the runners to finish executing the pipeline can take a while, often minutes for an enterprise pipeline. This creates long feedback loops, reduces delivery velocity and falls short of the promises of DevOps.
The most mature platform teams will usually implement tests to ensure nothing breaks. They’ll use test projects and will trigger the pipeline on each of those and ensure a successful run. There is a huge problem with this: it doesn’t scale.
The number of possible project configuration variations goes up 2n where “n” is the number of binary conditions, switches or independent branches in the pipeline’s code. In practice, every supported technology, conditional statement, trigger or execution rule adds another decision point. As those decision points accumulate, the number of possible pipeline paths grows exponentially, making it unrealistic to test every outcome of this pipeline.
As an example, this average enterprise pipeline has 66 “if” statements, which means it would require roughly 73 quintillion independent test cases to test every possible outcome. This is far beyond what a platform team could test exhaustively. Even maintaining a hundred test projects would already be costly and impractical.
In one real-life example, a company had approximately 1,000 test projects for 1,000 test cases. Each project was testing a different path their CI/CD templates might take once executed based on expectations of how developers would use their templates. This required a runner farm of 24 runners and an entire day to run once.
This is because a key limitation of CI/CD pipelines is that they usually need to be run so they can be fully tested. Although there exist tools such as act which can help reduce the amount of compute required, their behavior doesn’t always match that of the CI engine they seek to emulate, and the compute required to achieve this amount of testing is still very expensive. It’s so time-consuming that, like in the previous example, the platform team might only be able to test one change per day. This is unsustainable.
Pipeline generation does not necessarily involve AI, despite what the term might suggest today. There are companies that have been generating their pipelines programmatically even before LLMs appeared.
The generation process is the following:
I’ve presented this approach in live demos at Arctiq’s first-ever DevOps Day event in Montreal on June 10th.
People found the generation approach promising. The approach resonated because it addresses maintainability and testability at the same time.
Organizations that have adopted it are already seeing the benefits:
On GitLab, pipelines can be dynamically executed at run-time, allowing for just-in-time execution of arbitrary CI/CD code. The following demo will show that aspect of dynamically generated GitLab pipelines.
The GitHub approach is less dynamic as GitHub Actions doesn’t support modifying pipelines at run-time. The approach that was used is instead a self-modifying pipeline.
The following demo illustrates how CI/CD pipelines can be self-modifying on GitHub.
A logical next step is to remove the YAML translation layer entirely.
Since YAML is being generated for a specific CI engine, we should ask ourselves if relying on a specific CI engine is still needed or whether it can be removed.
We can actually do away with it entirely, and instead allow a general-purpose programming language like Python, Go, TypeScript or Java to run the pipeline. We would thus be merging the “template engine” piece of our generator into the CI platform’s runner, making our templated pipelines truly independent from any CI engine.
The architecture of the custom template-based runner I’ve created is very simple:
In the following demo, I’ve implemented my custom runner using Railpack and Paketo Buildpacks, two open-source tools that detect the requirements of the application stored in my Git repository and generate the build and test steps for them which are executed through Docker.
Generated pipelines are a strong answer once template-based reuse becomes too complex to maintain. This approach is still uncommon, but it’s not entirely new. Some engineering organizations have already implemented this approach to CI/CD, saving significant engineering time.
It’s a paradigm shift: Pipeline generators treat pipelines as a software engineering problem rather than a YAML maintenance problem.
We have seen similar approaches emerging in other products:
These tools point in the same direction: as delivery systems become more complex, declarative configuration alone is not always enough. If we can programmatically define container builds, CI workflows, and cloud infrastructure, then CI/CD pipelines are a natural next step for automation.
Arctiq helps organizations design and implement modern platform engineering frameworks, CI/CD pipeline modernization, developer workflow automation, and DevSecOps enablement that improve developer experience while accelerating secure, reliable software delivery. If you're ready to modernize your software delivery platform, connect with our team to get started.