Automating Container Base Image Updates Across Multiple Projects in GitLab

The Evolution of DevOps Platforms: Stay Informed Before Customizing 

In the fast-paced world of DevOps tools, platforms like GitLab are constantly evolving, frequently adding new native features that may eliminate the need for custom solutions. Before diving into custom development — which can be both tempting and powerful — it's crucial to thoroughly check the latest documentation to ensure you're not reinventing functionality that's already built into the platform. 

This blog post explores how we approached the challenge of automating base image updates across multiple projects using GitLab's extension capabilities, while emphasizing a key principle: always verify what's natively supported before building custom solutions. 

The Challenge: Base Image Updates at Scale 

If you're managing multiple applications that depend on the same base container images (i.e, microservices architecture), you're likely familiar with this scenario: a new base image version is released with critical security patches or performance improvements, and now dozens of dependent applications need to be updated, tested, and redeployed. 

Manually updating each project is not only tedious but error-prone and inconsistent. What starts as a simple update can quickly consume days of engineering time, delaying the rollout of important security patches and creating unnecessary technical debt. 

At Arctiq, we've tackled this challenge by leveraging GitLab's powerful APIs and webhooks to create a custom automated solution that drastically reduces update times and ensures consistency across projects. 

While we'll showcase base image updates in this example, the principles and approaches we discuss can be applied to numerous other cross-project automation scenarios. 

Understanding Base Image Update Cycles 

Before diving into automation strategies, it's important to understand typical base image update patterns. According to Red Hat's documentation on base image vulnerability tracking: 

• Standard release cadence: Base images like UBI are typically built on a standardized 6-week release cycle 

• Emergency security patches: Critical vulnerabilities trigger out-of-band updates outside the normal schedule 

• Predictable versioning: Most base image providers use semantic versioning to communicate the significance of changes 

This predictable pattern creates both challenges and opportunities: while frequent updates increase security maintenance overhead, their regularity makes them ideal candidates for automation.

From Polling to Push Notifications 

Before arriving at our webhook-based solution, we went through an earlier implementation phase that's worth discussing as it highlights important design considerations. 

Initial Approach - The Polling Method 

Our first attempt used a polling approach: when a pipeline ran for any target project, it would check if a newer version of the base image was available. This check was implemented as a consistent, maintained part of our build system libraries. 

Limitations of Polling 

While this solution worked, it had significant drawbacks: 

1. Inconsistent Updates: Active projects that ran pipelines frequently would pick up updates quickly, while less active projects might never update until manually triggered. 
   
   
2. Update Verification Challenge: We had no easy way to verify which projects were using outdated base images across our entire ecosystem.
   
   
3. Release Coordination: When preparing a new release version, we needed to ensure all dependent components were using the latest recommended image version - a manual and error-prone process. 

Moving to a Push-Based Approach 

These limitations led us to develop the webhook-based solution described earlier, which actively notifies and updates all dependent projects, regardless of their activity level. This ensures consistent base image versions across our entire ecosystem and simplifies release coordination.

When to Extend vs. When to Use Native Features 

Before diving into our implementation, it's worth emphasizing a critical approach we follow at Arctiq: 

1. Check Documentation First: GitLab and other DevOps platforms evolve rapidly. What required custom code last year might be a simple configuration option today.
   
   
2. Stay Updated: Subscribe to release notes and changelog updates for your critical platforms.  
   
   
3. Evaluate Native Solutions: When facing a challenge, first evaluate if it can be solved with native features before investing in custom development. 
   
   
4. Consider Long-term Maintenance: Custom solutions require ongoing maintenance as the underlying platform evolves. 

For our base image update challenge, we reviewed GitLab's capabilities before deciding that an API/webhook-based approach was the most appropriate solution for our specific requirements. 

Push-Based Automated Cross-Project Updates 

Our final approach leveraged GitLab's group webhooks and APIs to create a fully automated pipeline that: 

1. Detects when a base image is updated (via tag creation event) 
   
   
2. Notifies all dependent projects, regardless of their activity level 
   
   
3. Creates branches and updates configuration files automatically 
   
   
4. Runs tests against the new image version to verify compatibility
   
   
5. Creates merge requests if tests pass 

This automation shifts what was previously a multi-day manual process into a 30-minute automated workflow, ensuring that security patches and updates are applied consistently and quickly across your entire application portfolio. Most importantly, it guarantees that all projects are updated - not just the active ones - which is critical for release coordination and security compliance.

The Code: A Closer Look  

Here's how we update configuration files across multiple projects using GitLab's API: 

While this implementation uses relatively simple API calls, it's important to note that GitLab frequently adds new features. Always check the latest documentation to see if there might be newer, more streamlined approaches to achieving similar results. 

The Technical Implementation 

The automation flow begins when a new tag is pushed to your base image repository: 

Step 1: Base Image Update 

When a new version of your base image is tagged (e.g., v1.2.3), this triggers the base image project's CI/CD pipeline to build and publish the updated container. 

Step 2: Group Webhook Trigger

Once the base image is published, a group webhook is triggered. This webhook notifies all projects within the specified GitLab group and its subgroups about the update. 

Step 3: Automated Updates in Dependent Projects 

For each notified project where appropriate, our automation: 

• Creates a new branch 

• Updates the project's configuration file with the new base image version 

• Runs the project's test suite against the updated configuration 

• Creates a merge request if all tests pass 

Step 4: Release Verification 

With this system in place, we can now easily verify that all components in an upcoming release are using the latest approved base image versions. Before a release, we simply: 

1. Generate a report of all projects and their current base image versions
   
   
2. Identify any outliers using outdated versions 
   
   
3. Ensure all pending merge requests are reviewed and merged 

This verification step, which was almost impossible with our previous polling-based approach, is now straightforward and reliable. 

Practical Considerations 

Our journey from polling to push-based notifications taught us several valuable lessons about extending GitLab at scale: 

1. Choose the Right Trigger Model

The polling model (having projects check for updates) seems simpler but can leave projects behind. A push model (actively notifying projects of updates) ensures comprehensive coverage but requires more complex implementation. 

2. Infrastructure Load Management

When triggering updates across many projects simultaneously, consider the impact on your GitLab infrastructure. Random delays, queuing systems, or batch processing can help prevent overwhelming your runners. 

3. Periodic Reevaluation

Schedule regular reviews of your custom solutions against GitLab's evolving feature set to determine if native functionality has made your custom code obsolete. 

The Results: From Days to Minutes with Comprehensive Coverage 

By evolving from a polling-based to a push-based approach, we've achieved significant improvements: 

• Time Reduction: Update process reduced from 2 days to 30 minutes 

• Quality Assurance: 100% test coverage for base image changes 

• Security Improvements: Critical patches applied consistently and promptly 

• Engineering Efficiency: Teams freed from repetitive manual updates 

• Comprehensive Coverage: All projects updated, not just active ones 

• Release Confidence: Ability to verify base image versions across the entire ecosystem before releases 

• Scalability: Solution works even with hundreds of dependent projects thanks to load management techniques 

Beyond Base Images 

The pattern we've established for base image updates can be applied to numerous other scenarios: 

• Dependency version management across multiple repositories 

• Standardized policy enforcement for compliance requirements 

• Coordinated releases across interconnected services 

• Cross-system synchronization with tools like Jira, ServiceNow, or Slack 

For each of these applications, we recommend first checking whether GitLab has since added native support for these workflows, as the platform is constantly evolving. 

References 

1. GitLab Documentation Home - Always check here first for native features 
   
   
2. GitLab Release Notes - Stay updated on new capabilities 
   
   
3. GitLab Webhooks Documentation 
   
   
4. Group Webhooks 
   
   
5. GitLab Repository File API 
   
   
6. Merge Request API 

Need Expert Guidance? 

At Arctiq, we specialize in helping organizations implement DevOps best practices and optimization strategies. Our approach combines: 

1. Deep Platform Knowledge: We stay current with the latest GitLab features and capabilities 
   
   
2. Custom Solutions When Needed: We build extensions only when native features don't suffice
    
3. Documentation-First Mentality: We emphasize using built-in capabilities whenever possible 

Our commitment to these principles and deep GitLab expertise are formally recognized. Arctiq is a GitLab Select Partner, delivering specialized services to ensure client success. We have also earned the Professional Services Partner (PSP) designation with GitLab, highlighting our proven ability to help customers plan, implement, and optimize their DevSecOps practices. More details about our GitLab partnership are available here. 

With our extensive experience across Git-based platforms, public clouds, CI/CD, and system integrations, we can help you navigate the rapidly evolving landscape of DevOps tools - knowing when to leverage native features and when custom solutions truly add value. 

Ready to optimize your DevOps processes? Contact us today to discuss your specific needs and discover how we can help you achieve greater efficiency and automation.