CI Pipelines and DevOps Assembly Lines - an analogy

Our need for faster application delivery has brought forth efficient ways to develop and collaborate. DevOps is at the epitome of this innovation curve with its agile practices. One such practice that is helping DevOps achieve its ultimate goal of better planning, development, testing, collaboration, and delivery is Continuous Integration. CI has enabled development teams to integrate their code advancements frequently. These frequent integrations mean frequent error checks & corrections and the result is efficient integration. However, the major challenge is our need for CI, CD, and parallel-forked staging has introduced us to more efficient but little complex workflows along with terms such as CI pipelines and assembly lines.

Being a developer, I wanted to know what lies underneath the hood. Here are some of my findings that might help us all:

CI Pipeline - a pipeline to continuous-successful results

Continuous Integration (CI) is a process that helps you to build, package, and integrate your application easily and frequently without any delays. It is a software development technique that integrates the code multiple times, thus making the process of coding trouble-free for all the developers. CI Pipelines go one step ahead and provide a path for even the smallest of changes made to the code from the development to the delivery cycle. We can create daily builds instantly in an automated way using relevant tools. So, how does a CI pipeline differ from a classic CI?

A classic CI has the following stages:

Now as our requirements gradually progress towards bigger goals, we can play around with these stages, making the CI a parallel one and/or a forked one. Parallel one saves us from writing multiple jobs for phases in the same stage, while forked one helps in branching the CI logic. We can decide the size of the pipeline, write jobs for each stage and modify it on the go. But in any case, we all have to do one important thing, viz., maintaining it for efficient functioning. So, how do we do it? Here are some easy steps that we can follow:

Follow a “no broken windows’’ policy.
Set up the pipeline with multiple stages so that it is easy to run the important/primary tests first. Run the security and unit tests first, then run the API and UI tests.
Try to provide the same environment for the next pipeline to run to ensure reliability.

Altogether, a CI pipeline operates effectively if it is fast, precise and reliable. CI pipeline is a perfect way to integrate changes continuously during the development process, and it will cater to the agile requirements at just the right time. In each build, the CI pipeline reduces the rate of risks, and the final product reaches the customer faster. Apart from this, here are some of the benefits of a CI pipeline:

Efficient monitoring for individual components of the pipeline: Developers can focus on writing code and simultaneously monitor the behavior of the system in production.
Test early, test often with automation: Easily identify bugs at an early stage without investing much time with automated testing tools and by shifting the testing toward left in the development process.
Swift feedback & efficient response: Once the code is committed the tests corresponding to it run concurrently and we can easily avoid any link breakage.
The version control: This option allows us to keep track of all the versions and easily access them, and we always have the option to roll back in case things do not go as planned.

Assembly Lines- the future of DevOps

CI is an efficient way of continuously integrating changes. However, sometimes, it is not adequately able to deliver the expected level of integration. And there are some aspects of development, like configuration management, which CI won’t be able to help us with. So, to solve this problem we can use different tools and platforms that can solve not only development teams’ problems but also test, implementation, security, and infra teams’ problems too.

However, there is one hurdle with this. All these tools and platforms could be from different organizations, and making them work in sync could be a different task altogether. DevOps assembly lines make sure that these distinct tasks are automated in a connected way. A DevOps assembly line is similar to an assembly line in an automobile factory. In our case, the vehicle is our application. It comprises separate stages for the software development process, and we can easily move the code to production without any hassle. In the classic approach to a CI process, it takes around five to ten minutes for a single process to complete, and then deployment would take another hour or more to finish which includes all the stages from development to production. Assembly lines make sure the activities performed by the CI, Ops, testing, security and deployment teams are automated and synchronized. But how did assembly lines get to this stage? Here is how the evolution happened:

And here are some of the steps that we can follow in our DevOps assembly lines:

Efficient connection between the automated islands
Declarative workflows
End-to-end monitoring

If you are still wondering why we should move from a perfectly fine working CI pipeline to DevOps assembly lines, here are some of the benefits of assembly lines -

Easy workflows and integration: With the help of the DevOps assembly line, we can easily establish workflows across various pipelines. It integrates smoothly with the source control systems, DevOps tools, repository management, and artifacts management tools and helps us achieve collaboration amongst tasks and teams. And at the same time, it opens possibilities to automate and scale.
Configurable notifications, transparency, and monitoring: Each stage within a pipeline can have configurable notifications. We can access each pipeline and the stages within it, along with the versioned data, logs, and status. Furthermore, roles and permissions can help us monitor the assembly line in a better way.
DevOps infrastructure backed: A DevOps assembly line can help us look after the entire DevOps infrastructure.

DevOps assembly line works as a standardized function across tools, silos, and application types. And it will connect these standardized elements to form an end-to-end process to effectuate absolute visibility, traceability, and auditability.

In short, CI pipeline is a more efficiently modified classic CI that makes sure the integration is continuous. Whereas the DevOps assembly line is an amalgamation of several such pipelines for different activities involved in the overall application development & deployment process. These activities could mean CI pipeline, infrastructure management, configuration management, test automation, Security practices, release management, etc. Assembly Lines enable us to accommodate heterogeneous activities to easily organize them in integrating our software development cycle for a measurable workflow. Is 2020 going to be the year that will see the shift toward assembly lines? CI pipelines are an integral part of our DevOps practices, and we can see them combined with other aspects to form a DevOps assembly line. According to a report by 2019 Global Developer, the percentage of developers who either deploy on-demand or more than once in a day is not even half. It comes up to only 43%, which is quite low. The percentage itself indicates that it is necessary to integrate a reliable CI pipeline and DevOps assembly line in the development process.