The Bakery Model for Building Container Images and Microservices – InApps 2022

One of the fundamental ingredients in the adoption of microservices and containers is the notion of immutable changes being packaged in the form of container images. This container image format leverages the transport system of continuous delivery (CD), and allows a progressively better model than virtual machine (VM) images or Amazon Machine Images (AMI), in AWS parlance. The container image format is the new executable or build asset that is built once, and then can run on any compatible microservices infrastructure.

The adoption of machine images as the transportable entity is often attributed to the rise of automation strategies that allowed development teams to produce the image as the deployment artifact. This began with the AWS AMI format, and since then a variety of tools have appeared in the ecosystem that eased the life of developers by promising a quick, reliable and easy way to generate redistributable machine images. This also opened up a debate around what’s the best way to distribute the application on infrastructure: whether to conceal all aspects of a machine and code into a factory made image that remains immutable, or to allow a certain degree of self-awareness and self-configuration.

One of the approaches to build the machine image is to bundle it with the application code as the output of a build process. The image is configurable to the extent that application-specific parameters can be passed to it, when a container instance is instantiated. These parameters could be used for service discovery and other application-specific toggles. This approach dictates rebuilding and redeploying the said image whenever any change is required, either to the application or to the application topology.

[cycloneslider id=”ebook-3-sponsors”]

The other approach of building machine images insists on creating a minimal machine image that is built only when there is a change in infrastructure, runtime or topology. For most code changes, new machine images are not rebuilt. New container instances are spawned from pre-existing machine images, and code is then deployed over it, either through boot scripts or via agents running on the machine instance. This approach prevents the need for continuous generation of new machine images and instead lets the newly created container instance discover and fetch the changes itself. This mandates autonomous behavior on the part of the container instances.

The difference between approaches, however, lies in the scope of what goes inside the build process of machine images.

It is critical to note that in both approaches, the common aspect is to keep creating new instances of infrastructure when propagating changes, instead of relying on a long running infrastructure footprint. The difference between approaches, however, lies in the scope of what goes inside the build process of machine images.

Netflix documented its internal process of creating baked machine images and open sourced the Aminator tool a while back. Tools, like Packer from Hashicorp, have become commonplace in the modern delivery pipeline. Major PaaS vendors, like IBM Containers, have entire toolsets for managing and creating images in containerized environments. Ventures, like CloudNative.io, which offer organizations a complete AMI lifecycle management tool called Bakery, are pitching this capability similar to the practice at Netflix.

Machine images have new meaning beyond their usual reference to VM images like virtual desktop infrastructure (VDI) or AMI. Container image is the new addition to the meaning of machine image. This addition is drastically faster, more lightweight, and is making more headway into the workflows of ordinary developers. The container image is more pronounced with developers compared to the predecessors, such as machine images, that were primarily part of the machine-based operations toolkit.

Defining the Model of a Bakery

A bakery is a form of infrastructure entity that embodies the process of acquiring, building and releasing machine images to allow repeatable deployments of working code. The output of a bakery is a baked image that is used to spin off instances of machines (VMs or containers) in any compatible infrastructure. The compatible infrastructure represents the required environment in the form of hypervisors or container engines that support the deployment of these baked images.

A form of bakery excels when its output of baked images is compatible to a varied type of infrastructure, reducing the need for bespoke images built for each type. The rise of containers represents the form of standardization that would make container-oriented bakery models more meaningful and widespread.

In any microservices and containers story, the production and maintenance of container images holds an important place. Hence, successful microservices and container adoption is not without elegant handling of the various facets of container images, including:

• Selection of base images.
• Image sharing and isolation across multitenants/multiple projects.
• Container image validation and verification.
• Container image workflow.
• Testing container images.
• Container image storage and lifecycle.
• Container image expiration and depreciation.

All this is integral to the bakery strategy and implementation in an organization.

Creating Global and Local Bakery Standards

The story of the bakery model begins with the selection of the appropriate base images that are inducted in the development process. Most likely, the selection of these base images will be based on the size of the image and the trust for distribution. Alpine Linux, and images originating from that continues to be a predominant choice for base images. It is important to note that each base image selected must be accompanied by a documented history to ensure that the origin of the image can be validated.

In an organization with multiple projects and varied teams, it could be detrimental to have each team select their own base image. Building an array of trusted base images that could be shared and standardized is an approach that could work for organizations at scale. A Global Bakery is a logical entity in an organization that focuses on standardization, governance and the lifecycle of base container images.

[cycloneslider id=”ebook-3-sponsors”]

These shared base images could then be further customized to each team’s requirements and inducted into the development workflow. From the team’s point of view, the focus is less about inter-team sharing and more about making the container image useful as the deployable base. I call this concept the Local Bakery — an entity that inherits the base images from the Global Bakery counterpart and is required to work at the scale of an individual team.

A workflow across various systems involved in the Global Bakery and Local Bakery models, and demonstrating how both the continuous delivery process and the container engine pull from the workflow.

The output of the Global Bakery is a standard pool of base images, each of which has been verified, to add to the base image registry. The images are then ready to be used by all teams in a service-oriented organization.

The output of the Local Bakery is the base image that is customizable to the individual service stack and deployable for the service. Both Global and Local Bakeries use continuous integration (CI) to build and test the container images before shipping (or deploying) them to their registry.

A simple tagging strategy is useful to ensure container images are easily searchable and identifiable. Usually, the build ID generated by the CI server is used as the basis to arrive at the tag. Both Global and Local Bakery entities could use this premise to avoid confusion of the “latest” tag.

The scope of features that go into a base image during the bakery model process, including the ability to facilitate multiple Local Bakery teams.

Discovering and Implementing Bakery Model Practices

Testing and Security with Images

Testing base images is crucial to the success of container adoption. Base images will get propagated across the entire infrastructure and will be used for further customization and extension by developers. Behavior-driven development (BDD) can be employed to test automated infrastructure.

By using a service like Serverspec, you can create RSpec tests to check if a given server is configured correctly. Serverspec scripts can be run against the container instances spawned from base images during the CI process. This form of script allows verification against a prescribed set of configured states that a container has when it’s run.

Penetration Testing and Security Assessment

The base images and the workflow surrounding them provide a good foundation to conduct an automated security assessment. Tools, such as Twistlock, and not-yet-public initiatives, like Docker’s Project Nautilus, can integrate with the CI and deployment pipelines for baked images and can perform continuous scanning to ensure the images are secured. If a vulnerability is found and fixed, a new base image can be generated from the Global Bakery, and then passed on to the various Local Bakeries. Each Local Bakery then leverages continuous delivery (CD) pipelines to release new containers based on the changed base image. This allows rapid remediation to be in effect as soon as a fix is introduced in the base images.

Avoiding the Need to Bake in Secrets

One commonly recurring practice for production deployment of containers is the storage of secrets, like credentials and auth keys, needed by the service running inside the container instance. A bakery can instill a common practice to manage the secrets that a service running inside the container needs from time to time. As part of the governance process, a Global Bakery can integrate key management solutions, like Hashicorp’s Vault or Square’s Keywhiz, into the base images, thereby providing a common foundation for all container images that are created further down the food chain. This will also remove the need for reinventing the wheel for each service team that runs its own Local Bakery; it allows correct practice of storing and accessing secrets to become institutionalized.

Configuration Management in the Bakery Model

Prior to the rise of mass adoption of containers, configuration management (CM) tools like Ansible, Chef and Puppet helped with converging a given set of machines to a desired configuration. However, with the advent of container images and their extremely small footprints, these assumptions provide little value to the cause of CM tools. Since container images need to be lightweight, a heavier CM approach sounds like an anti-pattern.

CM tools do, however, merit an ordered way of orchestrating installation. In some cases, images are being built with CM tools and are provisioned via a Chef-solo kind of mode, without the need for any external CM servers. The way these two paradigms will converge or diverge depends on the strategy that container engines take in the coming years with regards to configuration management.

Global Bakery Through Automation

As mentioned earlier, the Global Bakery is a logical entity and is usually used to generate shared base container images that can be adopted by each service team in an organization. The Global Bakery can be implemented as a set of automated tooling that exists between the Local Bakery and the public container images hosted on Docker Hub, Quay.io, IBM Containers, or other registry services. An automated Global Bakery must perform the following in an organization setting:

• Validate base images from the origin to check if a valid history exists.
• Compare alternatives for the base image based on the size of the image and community strength (this refers to the number of people using the image from Docker pulls).
• Perform a security assessment and vulnerability scan.
• Create a new resultant container manifest (Dockerfile) that will install management tooling for debuggability, monitoring, log stream and secret store, and so on.
• Add metadata to the resultant base image in the container manifest.
• Push the resultant container manifest to the versioning system and the generated image to the registry, the Global Bakery store. This is used further by individual teams, each running their own Local Bakery.

Retiring Old Base Images

A base image goes through multiple versions over a lifetime, and eventually is discarded from usage. Release of new versions of a particular global base image can be communicated to individual teams so that changes could be made to the local images. Alternatively, the CI infrastructure for each service team may check for new versions of the base image on each iteration, and can trigger a warning if an older version is still operational.

The Way Ahead for the Bakery Model

The bakery model is potentially a very important entity in the infrastructure for microservices. There are new initiatives, like improved container workflow around unikernels and changes to Docker image IDs, that promise to make the bakery model even more useful and relevant. If effectively implemented, a bakery can allow service teams to become more independent and deliver on the promise of microservices.

Feature image via Pixy.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.