Building a DevOps Practice

DevOps is a culture where people, regardless of title or background, collaborate to develop, deploy and operate a system. It encompasses a variety of domains and best practices including QA, software development and operations. Success depends upon collaboration and a shared mindset where cross functional teams consider the implications of delivering software into production throughout the software development and deployment life-cycles.

In order to achieve DevOps success, it's important to articulate the current state of software deployment operations. What teams are involved, what tools are used, what's performed manually and what's automated, what's the current process, what measures are used to declare software deployment victory, …? Once deployed, how is the production environment monitored?

One way to describe the current state is to look at the core components of the DevOps practice and build a diagram. Include source control, build systems, deployment systems, environments, DC health monitoring, code level instrumentation, logging and APM.

Color code this diagram - red, yellow, green - based on the level of improvement required. Update the diagram, on an ongoing basis,  and include detailed notes during the life of the project to understand which pieces are doing well and where help is indicated. The team should review this regularly as a healthy DevOps practice will grow and adjust to the changing needs of the organization.

Associated with this is a clear understanding of how teams currently plan their work. Are tasks accomplished in isolation? How are tasks estimated? What tools are involved? What is the general workflow? Is there an approval process? How are issues identified, tracked and remediated?

With a granular view of the current state, teams can then start to consider what they want the end the end environment to like. This will encompass a lot of decisions and debate so the process must be well managed and facilitated.

There are two pillars to improving DevOps in any organization. The first is defining a release progression pipeline. This is a path that covers how the team controls source, manages environments, turns code into deployable artifacts, deploys and tests those artifacts in each environment and handles creation and management of those environments. It involves managing continuous integration, QA, staging environments and the push to production as a cohesive whole. Automation is a critical component of this process and should be implemented in each step as appropriate.

The second pillar is instrumentation. This covers the metrics obtained from the code once in production and how products enable troubleshooting through logging. This includes the ability to identify, analyze and remediate issues. Instrumentation is the key to maintaining a steady deployment-production state and many DevOps practices fail despite getting the release pipeline right.

Instrumentation is a broad subject with many interested constituents. Areas and functions worth measuring need to be defined in consultation with Product Managers, Product Owners and Production Engineering. Code should be instrumented so that the selected areas/functions can be measured for analysis and alerts generated  when the measurements fall outside expected norms (thresholds, trends, etc.)

Instrumentation is key to the success of any DevOps practice. Each product should-

1. Have environments that are well instrumented and monitored. 
2. Have code that is well instrumented and monitored so that key service level metrics can be accumulated in production. 
3. Log all activity, important, trivial or otherwise, to a central event log that is accessible by all. This log should send alerts based upon agreed to events exceeding their threshold. 
4. Have away of informing the team about customer experience. Performance and response time come to mind, for example. 

It's important to ensure that all exceptions are logged and analyzed. There should be a standard format for everything that is  logged to a file. A leveled or partitioned log should be implemented that distinguishes different message types. DEBUG, INFO, ERROR or CRITICAL, for example. Most Syslog and Eventlog formats provide strong, standard segmented formats. Each deployable product requires the same four components. These are-

1. Having code that resides within a single source control repository with a well known and documented branching scheme. 
2. Having a way of automatically building and testing the code with deployable artifacts as the end result. Note: Configuration details should be separate from the artifacts.
3. Being able to automatically deploy to different environments.
4. Having environments that can be automatically provisioned and configured.

It is best to have one repository per deployable product that acts as the primary source of truth for that product. It should is always be up to date. A single source supports rapid and safe, repeatable deployments. If properly managed, it also provides a history that makes it easy to see why changes were made, by who and when.

Many software products are built upon multiple stacks involving different technologies. While this adds complexity to the environment, it must be accounted for. From the DevOps perspective, the stack(s) and environments they run on must be able to:

• support automation 
• be managed through a regular, sustainable technology upgrade program
• support rapid, safe, continuous delivery 
• support an automated, repeatable way of provisioning environments for all uses (including production).

Release progression is centered around people who are domain experts in their disciplines coming together to address issues and solve problems across the pipeline. This includes version control, continuous integration, quality gates, approval and release to production. This workflow should be automated to the maximum extent possible, with exceptions being published as events and then managed.

Getting the release progression pipeline right is a non trivial exercise. It involves close collaboration with all of the development-deployment stakeholders. Expect bumps along the way and learn from mistakes. Similar to development generally, a successful DevOps practice is all about continuous improvement.

Teams should consider and define a DevOps baseline. That is, what needs to be implemented to see the benefits of the practice and what are those benefits in terms of the release pipeline, instrumentation and improvements.

Release Progression Pipeline:

• Automated deployments for all products across all data center locations. You don't need to get there, with every product, all at once but this is the goal. 
• Integrated QA environments allowing automated unit, functional and performance testing before the release of any product.
• Full deployment to production with “Release/Rollback” controlled by a simple load-balancer change. 

Instrumentation:

• Understanding the customer experience in detail. 
• Understanding application performance in detail.
• Dashboards that expose the available information across the organization.

Measurable improvement:

• Ability to track accepted DevOps Metrics 

DevOps metrics come in all shapes and sizes. The following are a few suggestions and a place to start.  Metrics should be tracked for all products both in terms of release automation and instrumentation.

Remember, if you don't measure it, you can't manage it.

• Deployment frequency
• Change volume
• Lead time
• Deployment success rate
• Mean time to recovery
• Customer ticket volume
• % change in user volume
• Availability
• Performance
• % instrumentation coverage

Note: A proxy measure for maturity of DevOps could be how frequently code is successfully released to production. Tools play a critical part in the process. They provide automation, alerting, metrics  and ease instrumentation.

There are many good tools to aid in the DevOps process. Determine the tools that best fit the environment and choose wisely. Part of the DevOps equation is infrastructure automation. It's not solely about the  continuous integration/delivery of software. Managing infrastructure through automation tends is often overlooked or considered a completely separate discipline. This need not be as both are key to the successful deployment and management of production software. 

DevOps is an initiative that provides a strong return on investment. While the tech team understands this, the same may not be true of the exec team. Given the expense in time, resources and dollars, it is important to spend time with the  exec team so they understand the value prop.

After all, teams need their support and the financial commitment that comes with it. Migrating from the current state to the future, automated DevOps state is complex. It must be well planned and supported. Time must be allocated for resources to work on the project.

This means that productivity will decrease for the tasks people are currently working on. Teams must define how they will accommodate the loss of available team hours. Equally important is for teams to consider how DevOps will impact their work and how the shift will be accommodated.

As a side note, executive sponsors will be keen to understand this and what the impact to delivery schedules will be. Be proactive in addressing this issue with sponsors.

All parties should understand the DevOps priority in relation to corporate objectives. Understanding and meeting expectations will pay dividends.  Since it is likely that productivity will decrease as each team transitions to the new DevOps process expectations should be managed for the team, stakeholders and the organization generally. Once the practices are in place and running smoothly, however, improvements in quality, time to deployment, error/rollback reductions, faster time to recovery, downtime reduction, team productivity and similar will be seen. Similar to any other initiative, building out a DevOps practice involves careful planning and clear communications. For example, discuss and then document the constraints and risks associated with the adoption of DevOps practices. Understand what groups and individuals are most impacted and ensure they're prepared for the requirements they'll be facing. DevOps is often associated with Agile practices. As such, run the project like any other. Define stories, track requirements, estimate workloads, track burndowns, run a daily scrum, have retrospectives and deliver working software after each sprint.

DevOps empowers teams to deliver software faster and safer. A practice with the right instrumentation helps teams identify and fix problems more efficiently. A practice with a proper release pipeline allows software to be deployed at any time and with ‘no downtime’.

A well formed DevOps practice allows teams to quickly find a problem and deploy a fix, often before customers even notice the issue. Finally, remember that migrating to DevOps is as much a culture and mindset shift as it is a technology one. An incremental approach to rolling out DevOps tends to work best.

Teams get there through careful consideration of the desired end state, education, collaboration, testing, tools and metrics.

Steve Rabin Chief Technology Officer, Product & Technology Center of Excellence More By The Author