In my previous post about remote and flexible work practices, I wrote about my own experience with documentation-centric collaboration. But how do you establish a documentation culture? How do you convince your team to embrace documentation in their daily work?

For this post, I thought I’d do something a bit different, and share some bite-sized suggestions for how to bootstrap and nurture a culture of documentation in your organization. These are all things I’ve tried with my teams, and found to be effective.

Here we go…

Create an employee manual and onboarding guide in your documentation system. This helps new employees get in the habit of looking at the documentation. And it creates a place that people will visit regularly to look up benefits, policies, and other employment-related content.

When someone asks you a question, document the answer, then respond with a link. This ensures it’s the last time you have to answer that question, and establishes a habit of looking at the documentation first. If you ask your team to follow the same principle, you quickly build up a library of fresh and relevant Q&A content.

Default to open. Whenever possible, make content available to anyone in the organization. You can’t predict who will discover your documentation and benefit from it — so why restrict your audience to just your own team or department? Plus, keeping things open creates an environment of transparency, accountability, and trust.

Highlight and celebrate examples of good documentation. You can show off great content in review meetings, weekly updates, or group chat channels. This shows you take documentation seriously, and creates a positive feedback loop for contributors.

Assume readers lack context. Don’t use jargon and acronyms without explaining them. Link to prerequisites, prior art, and other relevant documents. This makes your documentation friendlier and more accessible. It also makes it less daunting for new team members to contribute.

Include documentation in your career progression framework. As people move into more senior roles, it’s increasingly important they can clearly express ideas to their teams and leaders. By calling this out in the career progression framework, you create both an incentive and an expectation for everyone to actively improve their written communication skills.

Share your personal notes, unless they’re truly private. Most of us write things down to remember them for ourselves. But if you put in the extra effort to organize your notes and publish them, your whole team benefits. Plus, chances are this extra step helps you clarify your own thoughts, and retain the information better.

Create a public list of things that need to be documented. When you’re bootstrapping a documentation effort, people don’t always know where to start. If you share a wish list with the team, you may get help building out the content — and you quickly find out who likes to write.

Regularly delete out-of-date content. Stale content makes it harder to identify relevant information, and sends a message that the documentation isn’t actively maintained. This undermines your efforts to build a healthy documentation culture. One exception to this rule is any documentation you maintain for historical purposes, such as decision records.

Cover topics you want your team to care about. If you’re committed to improving something like diversity, make sure there’s documentation around it — such as guidelines for inclusive hiring practices. This is especially important if you’re a leader in your organization, because people consciously and subconsciously optimize for the things you focus on.

Document tactical and operational things, like project plans or coverage over holidays. This brings people into your documentation daily, and makes the content feel current and relevant. If possible, surface project status in your documentation as well, so stakeholders don’t have to dive into your task tracking tool to keep tabs on progress.

Create templates for interview notes, onboarding checklists, decision records, and other recurring content that should follow a specific format. Better still, create a quick link that generates a new document of the selected type, and populates the author, tags, and other metadata. This lowers the barrier of entry for creating structured documents, and encourages consistency.

Add a list of popular or recently edited pages to your documentation landing page. This gives new arrivals a way to sample the documentation contents, and helps everyone find documents the team is actively collaborating on. If you want to gamify your documentation with a bit of friendly competition, some documentation tools also let you add a list of the most prolific contributors.

Appoint an information architect to keep things clean and well organized. Low quality and poorly structured documentation turns people off. If you have the budget to hire a trained information architect, that’s amazing. But for everyone else, find someone in your team who cares deeply about documentation. You can ask them to spend a portion of their time organizing and maintaining your content. Perhaps they can also help evangelize your documentation culture, educate contributors, or create a house style guide. If you run a team and you have no takers, then congratulations, you’re it!

Good documentation is a superpower for distributed teams. Hopefully these ideas get you thinking about new ways to support the documentation culture in your organization. And if you had success with other approaches, I’d love to hear what worked for you.

My team at Cornerstone OnDemand went all in on remote work in the early days of the pandemic. But we didn’t have much prior experience working remotely, so we had to trial new ways to collaborate and get things done.

Here, I’ll share some of our successful remote work experiments … and a couple of failed ones, too.

Experiments That Worked

Full Schedule Flexibility

When we first switched to remote work, we kicked off another experiment at the same time: Total schedule flexibility. We asked people to work 40 hours a week, on whatever schedule suited them best, and to coordinate as necessary with their teams. This was not a stretch for us, as Cornerstone was already a global organization, with engineers collaborating across North America, Europe, the Middle East, Asia, and the Pacific.

We ended up with a surprising diversity of work schedules. Engineers started work as early as 5am, or as late as 11am. Some preferred to work a “4x10” schedule, with four long days a week. And a few came up with more fragmented schedules, working in shorter chunks of time over five or six days. We weren’t surprised to learn this, but it was clear the typical 9-to-5 office hours are not a good fit for most people.

Allowing people to organize their work around their personal lives and daily energy levels was a big win for the team. I loved that we could support people who preferred to take an afternoon nap, or go mountain biking during daylight hours, or maybe offset a production incident on the weekend by spending Monday with the kids.

But for me, the most positive impact was something we didn’t really expect: Some parents with childcare responsibilities were able to go back to full time work, when a regular work week would have been impossible to manage. They designed their schedules around their children’s needs, getting their work in at odd times of day and night. This seemed like an amazing win from a diversity, equity, and inclusion perspective.

The team also embraced global time zone differences. With most of my team in New Zealand, and collaborating closely with teams in the US, our work week only had a four day overlap. This created natural quiet periods on Monday in New Zealand, and Friday in the US. We committed to making these meeting free days, to give teams at least one day a week for uninterrupted, deep work. And for engineers that preferred to work four long days a week, this created a full day where they could be absent without missing any meetings or social gatherings.

Focus on Documentation

Remote and flexible work calls for a more asynchronous approach to communication. You don’t always have others available to answer questions or give feedback, and it’s awkward to depend on meetings for decision making.

To combat these challenges, we trialled a more documentation-based approach to collaboration. One of the most successful experiments was to use “architecture decision record” (ADR) and “product decision record” (PDR) documents to help us make decisions.

For any significant decision, an engineer or product manager would document the problem statement, any context or constraints, a list of possible solutions, and a recommendation for what to do next. The document owner would then ask for feedback from their peers and other stakeholders, iterate on the proposal, and ultimately commit to a course of action. These documents enabled asynchronous decision making, helped us prep effectively for any discussion meetings, and recorded each decision for future reference.

In parallel, we built up a library of “how to” content. Whenever someone asked or answered a question, they would document the answer in our internal wiki. Over a few months, this built up a large library of knowledge for new or existing engineers to tap into, at any time of day or night.

We also used the wiki to share project status, team priorities, key metrics, wins and challenges, and video demos of new product features. Team members would update the content regularly, and interested parties would then consume the information when and where it was convenient for them. This was a great way to inform the team and leadership about current events, without the need for all-hands meetings or expensive leadership syncs.

Fewer, Better Meetings

With our shift to flexible work schedules and improved internal documentation, we cut down drastically on meetings. We eliminated many tactical update meetings, group decision making sessions, and other meetings that could be handled through asynchronous collaboration. Some teams experimented with moving daily standups to group chat. They would then organize a follow-up meeting when there were interesting topics to discuss.

This shift created much larger blocks of uninterrupted focus time for engineers. It also increased engagement in the few meetings we did have, because we only organized one when we really needed to, and participants weren’t fatigued from consecutive meetings.

Perhaps more importantly, we shifted the focus of our few large meetings from work discussions to social get-togethers. With the money we saved on office-based perks, we set up a regular schedule of team catchups for food and chill-out time, or team activities like bowling, escape rooms, and paintball. This helped offset the lack of daily face-to-face contact in our remote work environment. It turns out face-to-face meetings are better when you just meet for fun.

Experiments That Didn’t Work

Hybrid Work

For the first couple of years of remote work, we continued to maintain an office. We wanted to enable in-person collaboration, have our own space for social catchups, and support people who couldn’t easily work from home. Although we didn’t mandate time in the office for anyone, a couple of my engineering managers bribed their teams with food for a bi-weekly catchup.

In practice, we found that only a handful of people used the office on any given day. As our remote work practices improved, almost everyone preferred to work from home, or occasionally from overseas. We rarely had a critical mass of people in the office for in-person collaboration, so the benefits of coming in were outweighed by the drawbacks — such as lengthy commutes or childcare difficulties.

We briefly considered mandating one or two days in the office each week. But when you do this, you negate the biggest benefits of remote work, such as location-independent hiring.

Ultimately, we canceled our office lease, and offered teams the use of a shared office space instead.

No Official Commitment to Remote Work

Although we embraced the remote and flexible work experiment, we didn’t commit to support these work practices indefinitely. We wanted to give ourselves the option to go back to office-based work in the future.

In retrospect, this was a mistake. A huge benefit of remote and flexible work is that people can make major life changes, like moving to lifestyle or low-cost locations, or cancelling daycare to work flexibly around childcare responsibilities. But buying a property in another location is risky if your organization doesn’t have an official remote work policy.

We found the full value of remote and flexible work was only realized a couple of years in, when we finally made a firm commitment to support it indefinitely. I wish we hadn’t kept the team on tenterhooks for so long.

How We Measured

When you perform these sorts of organizational experiments, how do you measure what works and what doesn’t?

My best suggestion is to continuously monitor the metrics that matter most in your organization. You you can plot these over time to see how each experiment affects them.

In our case, we tracked our internal performance metrics across application delivery, process flow, operations, and customer experience. This included things like deployment frequency, cycle time, incoming defects, and product usage metrics. You can read more about these in my previous post about monthly report cards.

To our surprise, we saw very little change to any of these metrics as we transitioned to remote work. We would have loved to see a clear improvement in some areas, but we were still happy with this result: It showed that the team quickly adapted to our new ways of working.

We also measured our internal NPS, through a regular anonymous survey. And wow, did we see a change there. Six weeks into our fully remote work experiment, our internal NPS was up by over 30 points. Even though it eased back a few points once the novelty wore off, this is the largest increase I’ve seen in my career.

This sentiment was backed up by informal feedback from the team, and regular check-ins with managers and engineers. People really appreciated the freedom and flexibility of remote work, and many made meaningful lifestyle changes in response. It’s hard to imagine any other organizational change that could result in such a big quality of life improvement for the team.

But what does the path to continuous deployment look like, and how do you make the transition happen? This post is my attempt to answer those questions.

My team at Cornerstone OnDemand went from a handful of production deployments annually to more than 2,000. We automatically deployed code changes to production as soon as they were merged to trunk, often in under 20 minutes.

But switching to continuous deployment was not an overnight change for us. The team spent over a year building tools, adjusting processes, and establishing the culture required for successful continuous deployment. In the years since then, we continued to develop and refine our practices.

As with any large change, you don’t have to do everything at once. If you break down the process into a series of incremental steps, it’s easier to make progress, and you’ll start to see the benefit of these changes long before you reach the end goal.

So rather than describe what a continuous deployment environment looks like when you’re already doing it, I wanted to share the incremental changes you’ll likely need to implement, across several different areas. There’s a handy tool for just this sort of thing, and it’s called a “capability maturity model”.

Maturity Models

In a nutshell, a maturity model helps you evaluate your current performance in a specific area, and suggests what capabilities you should pick up next to improve.

You traditionally measure performance in five levels: Initial, Repeatable, Defined, Capable, and Optimized. Although if you prefer, you could simply think of them as Level 1 to Level 5. The key thing is that once you’ve identified what level you’re at, you should strive to reach the next level before you attempt to add capabilities from the higher levels. This gives you a blueprint for how to incrementally improve, until you achieve mastery in a specific area:

When it comes to complex initiatives — such as introducing continuous deployment — it’s rarely enough to measure your maturity in a single area. In these situations, you can evaluate maturity across multiple dimensions. It’s helpful to represent this information in a grid, where each row shows a dimension, and the columns indicate levels and the associated capabilities. Here’s an abstract representation of what this might look like:

A maturity model is a tool, not gospel: The idea here is to offer guidance for teams that want to make the transition to continuous deployment, but don’t know what concrete steps to take, or in what order. Depending on your specific situation or organization, your journey may be different — and your destination as well.

A Maturity Model for Continuous Deployment

My team at Cornerstone identified six key areas that we considered crucial for continuous deployment:

• Test automation is important because there is no latitude for manual deployment decisions with continuous deployment. Once you merge code it goes straight to production, and you need enough automated safeguards to ensure this will not break existing functionality, or introduce new issues. For more on this, see my previous post on test automation for continuous deployment.

• Observability matters because you’re constantly shipping code changes to production, and you need to find out immediately if something went wrong. With good observability tools, you can pick up anomalies in production as they happen, and link them directly to the corresponding code change. This allows you to quickly implement and ship a fix. You should also instrument internal tools to ensure you can measure and troubleshoot every part of the continuous deployment workflow.

• Deployment automation is at the core of the continuous deployment journey. Each engineer may be deploying to production several times a day, which means they use the deployment tooling a lot. The tools should be safe, robust, and automated to the point where engineers trust them, but don’t have to spend much time thinking about them.

• Engineering process changes allow you to safely deploy smaller increments of work. It’s particularly important to optimize for flow, to minimize the time between code change and production deployment. And much of the benefit of continuous deployment will only be realized when the team measures the customer impact of each change, and uses this information to guide further development.

• Release process is a cross-functional concern, in that you need to engage stakeholders across the organization to establish working agreements for continuous deployment. To facilitate this, you need a mechanism to separate code deployments from product releases. If you work in a large company, my earlier post on continuous deployment in the enterprise offers some ideas for how to do this.

• Cultural and organizational change is key, but often the hardest to implement. In my experience, the most important change is to develop a culture of ownership, where engineers take end-to-end responsibility for everything they work on. This will be easier in an organization with integrated product management, engineering, QA, and operations teams.

Here’s my suggested continuous deployment maturity model, based on these dimensions:

If you prefer to view or save this in PDF form, here’s a download:

Continuous Deployment Maturity Model

62.4KB ∙ PDF file

Download

Download

Applying the Model

Whether you use this model or construct your own, there are a few things to keep in mind…

Ideally, you make progress on all these dimensions in parallel. But since this is not always practical, I’ve sorted the dimensions from top to bottom in the order that you should probably start. For example, you’ll benefit immediately from improved test automation and observability — but increased deployment automation will have limited impact if you don’t have the other two.

In this context, you may be surprised to see culture at the bottom. I’ve often heard objections to continuous deployment on the grounds that “we don’t have the right culture for it”. The assumption seems to be that you need to establish the right culture before you can start.

In my experience, this is entirely backwards: You can never build the right culture if your engineers don’t have the tools to safely do their job. Once you build out your automation capabilities, and ratchet up your processes to deploy more frequently, engineers will feel more empowered, and will naturally take on more ownership. Culture change will sprout from this, and you can help it along with some gentle nurturing.

Either way, it’s a good idea to at least achieve the “Initial” state in all dimensions before you start on continuous deployment. Without these fundamentals in place, increased deployment frequency will reduce safety and increase manual deployment work.

Here’s a quick animation of how we progressed through this framework with my team at Cornerstone OnDemand. As you can see, it was not a linear process — and we still had a few areas that needed work:

Approaching Done

Switching to continuous deployment is itself a continuous process. You build up the technical and cultural maturity over time, and continue to fine-tune it indefinitely. That makes it hard to pinpoint the moment when you’ve “arrived”.

For my team at Cornerstone OnDemand, the aha moment came when newly hired engineers were able to deploy a code change to production in their first day on the job. They would merge a change to trunk, watch the build tools automatically validate and deploy it, then test drive it in customer-facing environments just minutes later. This was an incredibly empowering experience for new hires, and brought home the level of automation and safety we had achieved in our deployment tooling.

The maturity model here is intended as a starting point and a guide. But the diversity of engineering organizations guarantees that every journey will be unique. So if you had a different experience introducing continuous deployment, or developed a different continuous deployment maturity model, I’d love to hear about it!

In retrospect, the engineering enablement teams were quite successful, and had a lot of organizational impact. They helped introduce continuous deployment, adopt DevOps practices, migrate to the public cloud, and continuously improve the developer experience. But they also ran into unexpected challenges that were not shared by their product engineering counterparts. This made me realize engineering enablement teams have different needs from their leadership.

If you’re starting a new enablement team, or taking over leadership of an existing one, chances are good you’ll face similar challenges. Here, I’ll share some of the things I learned, and suggest what you can do about them.

Spend Time on Marketing

Product managers don’t just help engineering teams decide what problems to solve. They also work hard to market their products internally and externally. They create awareness, drive adoption, author documentation, and support product launches. But if you run an engineering enablement team, that person is probably you.2

This means you have to get your work in front of an internal audience, and convince them to adopt it. Even if you’ve built something that clearly improves the engineering experience, people will be busy, and wary of change. And it’ll be even harder to get traction if you’re introducing new technologies or ideas that are unfamiliar. Perhaps you’re launching the first component library in your organization, or implementing a new test automation framework, or integrating security tools with your build pipeline? It won’t be enough to show the work: You also have to explain the benefit — repeatedly, in different forums — and help teams get over any barriers to adoption.

If you don’t market a solution enough to get broad adoption, you may find product engineering teams muddling along with painful but familiar processes — at great cost to the organization. For example, if your front-end tools team can’t get adoption for their standardized library stack, product engineers may just continue to manually update third party dependencies sporadically, each on their own schedule. It’s hard for teams to quantify this wasted effort, and hard to justify an upfront investment to achieve some benefit later.

Worse still, teams may come up with their own solutions to problems you’ve already handled. This results in confusion, incoherence, or outright internal competition.

Here are some ways you and your team can market your work internally:

• Show your work in all-hands meetings, weekly demos, communities of practice, and other forums. It’s rarely enough to show something once, so you need to keep reminding people of the products and services you provide.

• Regularly share internal blog posts, how-to videos, and other content to help engineers get started. Create great documentation, and make sure it can be found. Run hands-on workshops and Q&A sessions, and offer proactive support in Slack. All of this creates visibility, lowers the barrier of entry, and demonstrates long-term commitment to your work.

• Update internal self service tools to ensure new projects adopt your work by default, and existing ones get automatically migrated. You’ll have an easier time creating awareness and achieving adoption if you integrate directly with the tools engineers already use.

• Share success stories, and publicly celebrate teams that have achieved benefits from using your work. This creates a sense of excitement and pride around improving engineering practices, and helps build strong relationships with your internal customers.

• Find influential engineers and keen early adopters in the organization, and work with them on new capabilities. If you can identify a handful of true fans, they’ll enjoy being consulted, help improve your product, and spread the word over time. When you’re working on something big — like a design system — consider creating an official community around it.

As any startup founder will tell you, building a thing is only a small part of making it successful. You need to spend way more time than you think on marketing, education, and evangelism.

Seek Leadership Buy-In

Even if you’ve mastered the art of internal marketing, it’s hard to achieve broad adoption for things that require cultural change. Introducing DevOps practices is a great example, or adopting something like continuous deployment. These types of transitions require a long-term commitment, with ongoing investment in tooling, education, and support. To pull this off, you need to secure leadership buy-in for your vision.

When working with your leadership, you’ll need to explain what you’re doing, and why it’s important. You should also be clear about what help you’re asking for — whether it’s space in leadership presentations, a time commitment from product engineering teams, or something else. And ideally, you should make sure engineering incentives are lined up to support your work. For example, if you’re trying to improve operational maturity, you need to make sure the leadership team recognize and celebrate proactive, sustainable behaviors, rather than individual heroics when something falls over in production.

One thing you might not want to ask for is any sort of top-down directive to use your enablement services. This creates resentment among product engineers, and removes your incentive to build the best possible product for your internal customers. For better long-term outcomes, focus on securing investment and public support, and let your work stand on its own merits.

Also consider senior leaders in your product management and design groups. Many engineering enablement projects will be highly relevant to them — whether it’s developer tooling to build product features faster, component libraries to achieve a more consistent user experience, or instrumentation for better product analytics. If these stakeholders understand what you do, they can be amazing allies in the organization. Help them out by showing how your work has a positive impact, and by articulating how they can support you. In return, they’ll help you out by creating visibility and driving adoption.

Very few technical transitions happen successfully without leadership engagement. And leaders of all stripes will be less familiar with engineering enablement concerns than they are with the product your organization sells to customers. If you want to increase the success and impact of your engineering enablement teams, it’s your job to ensure the leadership team understand and agree with what you’re doing, and help you pull your projects forward.

Show the Team Their Work Matters

Engineering enablement teams can have amazing impact, especially in larger organizations. With hundreds of engineers, even small improvements in the developer workflow add up to meaningful gains. Also, like all craftspeople, engineers love their tools. They’re especially appreciative of automation, and robust platform services that abstract away concerns like consistent instrumentation, backup and restore, and management of security vulnerabilities in third-party dependencies.

Despite all of this, I’ve found engineering enablement teams often struggle with their sense of achievement. While product engineers get a regular dopamine hit every time they ship a new feature to customers, enablement teams create leverage — and leverage is hard to see or measure. As a result, enablement engineers fret about whether what they’re doing is really important, whether anyone values their contribution, or if they’ll become redundant when there’s “nothing left to improve”.

As an engineering enablement leader, you have to show the team how their work has value. Here are a few ways you can do this:

• Spend time on a clear vision and mission for the team. This will guide the team’s efforts in a manner that doesn’t have a specific end condition. It also helps generate and prioritize ideas for how to continuously improve the developer experience.

• Measure engineering effectiveness and efficiency across all product engineering teams. This helps you identify areas that need work, evaluate the outcomes of experiments, and track the impact you’re having over time. Making this data public can be a powerful motivator for your team. One way to do this is through a monthly report card.

• Measure and publicize usage of the tools you build. When you create tools to automate manual workflows, it’s often astonishing how much use they get — and hence how much time they save. Engineers and leaders love to see this kind of data.

• Share regular demos and retrospectives with the wider organization, perhaps on a monthly basis. This is a good way to celebrate your achievements, and highlight how they benefitted other engineering teams. It’s also a great opportunity to call out where the pain points are now, and what you’re planning to do about them.

• Offer interested engineers the opportunity to rotate between product engineering and engineering enablement teams, for a few months at a time. This builds empathy and appreciation between the different roles, and lets enablement engineers experience the impact of their work firsthand.

It’s super fun to run engineering enablement teams, but they need a different brand of leadership: You’re no longer just an engineering manager, but also a product manager, product marketer, evangelist, educator, and cheerleader. You need to build leadership support for your team, get word out about the work you do, and show your engineers how their efforts matter. If you do this, your team can amplify the impact of all engineers across the organization.

If you automate an existing manual regression test suite, you’ll end up with a preponderance of end-to-end tests that are expensive to maintain and slow to run. At the same time, this approach doesn’t put enough emphasis on non-functional concerns, like security, accessibility, performance, and coding style. None of this is conducive to successful continuous deployment, where you need to deploy quickly and frequently, with a high degree of confidence in your overall product quality.

Modern deployment practices call for a modern approach to test automation. To do continuous deployment well, you need a test suite with:

1. Broad test coverage…

2. …but not too deep

3. Zero tolerance for test failures

Let’s find out how these guidelines ease the transition to continuous deployment.

Broad Coverage

In software engineering organizations that ship monthly or quarterly releases, engineers tend to focus on fast feedback tests they can run locally. This includes unit tests and light integration tests. Maybe they also integrate their changes regularly, with automated functional testing on a continuous integration server.

Thorough regression testing happens at release time, along with security testing, performance testing, accessibility testing, and any other tests that are costly to set up or run. Very often, a separate team execute these tests, in a dedicated environment they own and govern. Product quality becomes a focus at the end of each release cycle, and teams may allocate time for “hardening”, to focus exclusively on testing and fixing bugs.

With continuous deployment, this model is no longer possible. Engineers deploy to production many times a day, and every deployment must meet the quality bar for production code. This means all forms of testing must be combined into an automated test suite, and run continuously, on every deployment. And the test suite must be broad enough to cover all functional and non-functional validation.

A robust test suite for continuous deployment will include unit testing, integration testing, and end-to-end testing. These should all be integrated directly into the deployment pipeline, and run automatically on every code change. But on top of that, any other tests you run occasionally today must now be integrated into the pipeline. This may include static code analysis, configuration checks, linters to maintain your house coding style, API contract tests, fuzz tests, and a battery of static and dynamic security tests. If you perform regular tests for accessibility and performance, you can look for tools to automate these as well.

Building out your automated test suite with these additional forms of coverage means you can enjoy production-grade release safety for every individual code change. With broad and continuous coverage, you’ll also reduce the churn that comes from fixing large numbers of non-functional test failures at the same time, during infrequent integration windows.

But Not Too Deep

Building up this level of test coverage seems daunting, but here’s the thing… You need broad coverage, but it doesn’t have to be deep. If the purpose of your test suite is to let you safely deploy code to production, then the ideal amount of coverage is literally as little as possible, while still achieving that goal.

With continuous deployment, that’s not a very high bar — because every change you deploy is small, and hence inherently safe. You’re no longer integrating a lot of features at the same time, and looking for regression issues across the entire product. Instead, you’re making individual, highly targeted changes in one product area at a time. These changes are easy to test locally, easy to reason about, and easy to peer review. They don’t need the same level of regression coverage you’d apply for a big bang release.

What constitutes enough coverage depends on the type of test. Third party tools for linting and static security testing come with hundreds of built-in rules that execute in seconds. You may as well run all of these, because they’ll help maintain product quality for virtually no effort on your part. Unit tests are cheap to build and run, and you can go to town on contract tests, configuration checks, and other cheap tests.

Instead, you should focus your efforts on tuning coverage for expensive tests, such as end-to-end tests and automated performance tests. The goal here is not to achieve comprehensive test coverage, but to exercise your most important workflows. These tests act as a sanity check, to catch anything that’s fundamentally broken in the product. Engineers will still test the specific areas they worked on before merging a code change. And you can use cheaper tests — such as snapshot tests — to catch any functional regressions in other product areas.

How do you know you’ve made the right tradeoff between test coverage, maintenance cost, and deployment speed? With some basic operational metrics, you can answer this question analytically. For example, if adding or removing tests doesn’t materially affect your rate of defects in production, you may have more tests than you need of that type. You can also set some high level goals around deployment speed and change failure rate. If you target a 15 minute lead time to production, and a change failure rate of less than 1%, your teams can tune their test suites to achieve those targets.

A less scientific approach is to measure your deployment confidence with the “Friday afternoon test”. If your engineers are comfortable deploying to production on Friday afternoon, that’s a good sign. They know the tests are fast enough and stable enough that they won’t waste their Friday evening getting the deployment into production. And they clearly feel the test coverage is good enough that they won’t get paged for regression issues over the weekend.

The concept of reducing test coverage seems alien at first. But to practice continuous deployment safely and sustainably, you need a good balance of broad test coverage, low maintenance costs, and fast execution times. That means you have to get comfortable trimming out tests that are not pulling their weight.

Zero Tolerance for Failure

Now that you’ve tuned your test coverage, there’s still the matter of making sure tests pass. With infrequent releases, much of the integration and testing work happens close to release time. This flurry of activity makes it inevitable that some tests fail. Humans need to review these test results, and make a judgement call on whether to fix the code, fix the test, or release regardless.

With continuous deployment, this is not practical. Even a small engineering team can deploy code to production a dozen times a day or more. When you deploy this often, there’s no room for human decision making. The tests have to pass every time — and if they don’t pass, you need to know it’s due to a legitimate defect. If you slow down to investigate a test failure manually, you’re not only holding up your own deployment, but your team will be piling up change sets behind you as well. There should be no manual decision points in a continuous deployment pipeline, which means you cannot accept test failures.

This is even more important if your organization is new to continuous deployment. You can expect some initial pushback around testing as people get used to the idea of fully automated deployments. Customers and compliance teams will feel more comfortable if you can guarantee that every deployment passed every test case.

It’s also easier culturally to enforce a 100% pass rate. Paraphrasing former Ikea Chief Sustainability Officer Steve Howard, if your target is 95%, everyone will find a reason they should be in the 5%. Engineers will become desensitized to flaky tests, and will just restart failed builds, or wait for others to investigate. But if you set the target at 100%, you create total clarity for the team, and engineers just get on with fixing every failure.

Besides, if a test fails, and you still feel it’s ok to deploy to production, do you really need that test? In the spirit of reducing the depth of coverage, sometimes the correct response to a broken or flaky test is to simply delete it. Adopting a zero tolerance policy for test failures helps keep your test suite small, fast, and dependable.

Get Started

With all this in mind, what should you do next?

It’s always a good idea to start with an assessment of your current test suite. If you haven’t adapted your test strategy for continuous deployment, your test coverage and test maintenance effort are probably heavily skewed towards functional regression tests.

The end goal is a broad test suite that’s fast, dependable, and easy to maintain. Trim expensive end-to-end tests down to the most important workflows — or those tests that most often find legitimate defects. Eliminate slow and flaky tests. Then add broader coverage as needed, to cover configuration, coding style, security, accessibility, and other non-functional concerns. And implement a zero tolerance policy for test failures, to improve consistency and deployment confidence — for yourself and for your customers.

If you haven’t yet adopted continuous deployment because you don’t have enough test coverage … are you sure? Continuous deployment is inherently safer, because each change is small and self-contained. That means you can probably get started with what you have. You’ll quickly discover where the gaps are, and you can tune the coverage incrementally over time.

In my most recent role, we muddled along for years with cascading goals that tied engineering performance to achieving top-level company objectives. Engineers had limited ability to influence these goals, and consequently didn’t show much interest in the goal setting process, or in measuring performance over time. This undermined our efforts to build a culture of ownership and accountability within the product engineering team.

This year, I got together with my product management counterpart Henry Vasquez to come up with a better framework for tracking our performance. What we landed on was a monthly report card with a broad spectrum of engineering and product metrics. We published the report card to our entire product engineering team, and made it publicly available to anyone in the business.

The report card ended up being more useful and relatable than our previous attempts at measuring performance. It created clarity for our cross-functional teams on what we considered important, and led the teams to independently tackle problem areas and advance our software delivery process. At the same time, the report card highlighted our successes and achievements, and gave us an effective tool to communicate with leaders and executives on our own terms.

Here, I’ll share how to create a monthly report card to spark a healthy conversation around product engineering performance, and build a culture of ownership and accountability within the team.

Choosing Your Metrics

The magic of a report card comes from including a broad spectrum of metrics. It’s difficult to define a single metric that aligns well with success for the organization as a whole. But by combining a range of product and engineering metrics, you can describe the kind of engineering environment you’re looking to establish.

The metrics you choose define how the rest of the organization view your performance, and send a strong message to the team of what behaviors and outcomes you value. If you want to focus the team on creating more value for customers, you need metrics to measure customer outcomes. If you care deeply about the wellbeing of your team, you should measure their happiness and their daily engineering experience. If you want to improve your operational maturity, you may want to measure your progress on infrastructure automation, or public cloud spend, or service reliability.

Ideally, some metrics will be thematically linked to your high-level company goals, and formulated in a way that’s relevant to product engineering teams. This creates clarity for engineers, while eliminating potential conflicts between engineering achievement and company goal achievement. At the same time, the report card should include metrics that you plan to track in perpetuity — such as deployment frequency and the rate of customer-reported defects. These evergreen metrics define what’s considered healthy in your daily work.

How many metrics you pick communicates whether you’re focused on achieving a small number of specific outcomes, or trying to build a well-rounded product engineering environment with fewer blind spots. If your startup is running out of money, it’s fine to focus on growing revenue, or reducing operational costs. But this sort of hyper-specialization will have unintended consequences over time. As your organization grows and matures, you’ll want to expand the scope of what you track. From our experience, 20–30 metrics from a wide range of areas gives you a good overview of your team’s health and performance.

With so many metrics in play, team members can easily succumb to metric fatigue. And tracking a large number of slow-moving metrics makes it difficult to get a quick read on whether you’re making progress towards your goals. So I suggest highlighting a subset of metrics that you’re actively trying to improve. You can do this by including targets for each metric. This shows which metrics you expect to move, and which ones you’re just keeping an eye on.

In our case, we assembled about 20 engineering metrics, and 10 product-centric metrics. These included:

• DORA metrics for application delivery; deployment frequency, lead time for changes, change failure rate, and time to restore service

• Flow metrics, like cycle time, pull request review time, and the average number of tasks in progress per engineer

• Operational metrics, like API success rate, 95th percentile response time, and AWS spend

• Incoming defects and product questions from customers

• Customer and user engagement metrics for key product areas, and tracking towards expected product outcomes

• Although not part of our report card, we also tracked employee and customer NPS scores on a different cadence

Out of these, we were moving about five metrics aggressively forward. For the rest, we kept an eye on overall trends, and made opportunistic improvements throughout the year.

If you’re looking for more guidance on how to pick engineering performance metrics, Nicole Forsgren et al. have you covered with The SPACE of Developer Productivity. The authors argue convincingly for using a selection of metrics from five dimensions:

• Satisfaction and well-being

• Performance

• Activity

• Communication and collaboration

• Efficiency and flow

Creating Visibility

Now that you’ve assembled your metrics, you need to decide how to present them, to whom, and how often.

You can host the report card anywhere, so long as the data is easy to digest; it could be a wiki page, dashboard, or a shared document. If you collect internal metrics in a data platform — such as Splunk — you can build a report that automatically pulls the latest information. My team used a simple tabular representation on a wiki page, with color coding to indicate the health of each metric. But graphs or sparklines to show changes over time may be helpful. Another suggestion is to use a rolling one-year window for the metrics, so you’re not looking at just a single data point at the start of the year. The trend is usually more interesting than the current value.

What does matter is that you make the data accessible to anyone in the organization. You should share the same report card with your team, and with leaders, executives, and other stakeholders in the wider company. Public visibility creates alignment and clear internal and external accountability. It also gives team members a sense of ownership — both of their goal achievement, and of how their work is represented.

It’s good to create clarity on how your metrics are collected. You could do this by linking each metric in the report card to the underlying data source. This allows people to analyze the data independently, to draw their own conclusions, or to explore new angles on the same information. It also reduces the risk of bias through cherrypicked data — or the perception of bias where none exists.

You may wonder why I’m specifically recommending a monthly cadence for updating the report card. Why not refresh it continuously, or align it with your company-wide goal setting cycle?

We arrived at the monthly frequency by a process of elimination. Although we used quarterly company-level goals, the report card needs to be more frequent. Otherwise, by the time you discover you’re off track, it’s too late to do something about it. Meanwhile, weekly updates exhaust the team, and are often unexciting. Very few metrics that really matter move this fast. For us, monthly updates was the goldilocks frequency to capture progress on product adoption and engineering improvement initiatives.

Once you’ve published your monthly update, invest in a burst of publicity. Share the results wherever your team hangs out: In Slack, email, internal wiki, all hands meetings, or reports to the executive leadership. Bring it up in 1:1s and career development conversations as well, and in any other context where you talk about engineering performance. Continuous reinforcement ensures your team and leadership take the report card seriously, and look at the metrics for guidance in their daily work.

Don’t shy away from communicating failures, or metrics that aren’t moving. The intent is to create full transparency, and spark a productive discussion about what to do next. Public failures are a great opportunity to create interest in the process, and bring people along on the journey of fixing the underlying problems.

Putting the Report Card to Work

The monthly report card was well received by my teams. It created a balanced view of our product engineering performance, guided our investments, and helped us represent our work to the wider organization.

One of my favorite benefits of the monthly report card is that it highlights continuous, incremental progress on slow-moving metrics. Engineers may strive to shorten cycle times, or increase deployment frequency, or reduce customer-reported defects. But it’s very hard to see the fruits of these labors in realtime, which makes it thankless work. With the report card, you can visualize month-over-month or year-over-year improvements on these metrics, and show the impact of incremental changes over time.

By carefully selecting your metrics, you can also use the report card to address specific practical or political challenges. For example:

• It’s a great tool for creating alignment between engineering and product management teams, because it creates joint ownership of a set of performance metrics. By picking more or less aggressive targets, you can indicate the relative priority of investment in each area.

• If your organizational goals are too abstract for engineers to rally around, the report card can bridge the gap. By anchoring your metrics in the high-level goals, you create concrete and actionable ways for engineers to contribute to the organization’s strategic priorities.

• You can use the report card to prioritize work for engineering enablement and platform teams. These teams can use application delivery, flow, and operational metrics to identify problem areas and create tools to improve the engineering experience.

• If you’re held accountable externally for specific metrics — such as change failure rate, test coverage, or service level agreements — you can bake these into your report card. This meets the external reporting requirement, while shifting the conversation away from a small number of metrics to focus on a more holistic view of product engineering performance.

This versatility means you can adapt the report card to meet the needs of your team, and to evolve along with your organization.

What kind of environment do you strive to establish for your team, and what are the behaviors you’d like to reinforce? How would you like to be measured by your leadership? What metrics would you like to move forward this year? If you can answer these questions, you can build a report card to help shape the culture of your team — and to take control of the narrative around product engineering performance.

I disagree. You can absolutely do continuous deployment in enterprise environments. And just like in smaller businesses, it will lead to better outcomes for you and your customers. In my most recent role, my teams built software for some of the largest organizations in the world. Collectively, these companies used our products to support millions of employees globally — many of them working in compliance-driven fields. Despite this, my teams deployed to production more than 2,500 times in 2021, with dozens of deployments on busy days. We delivered a steady cadence of product improvements, with consistently high quality and no disruption to service. And we were able to turn around defect fixes for customers in as little as 25 minutes.

So why the angst about using continuous deployment in enterprise environments? The objections I’ve heard generally focus on three things enterprise software customers expect from their vendors:

• You need to meet stringent safety and security requirements

• You need SOC 2 and ISO 27001 compliance

• You need to communicate upcoming product changes well in advance, to avoid surprises

I’m here to tell you that continuous deployment is not only compatible with these requirements, but it helps you achieve them. Let’s take a look at how.

Safety & Security

Large companies have strict requirements for safety and security. And justifiably so: The legal, financial, and reputational impacts of security incidents can be huge. When these companies purchase software, they make sure their vendors have a robust and multi-layered security strategy in place, to protect against internal and external threats. Your software delivery process has a key part to play here, and this is where continuous deployment helps in several specific ways.

First, you can integrate many security tools directly into your development workflow. This includes tools for static security testing, application configuration checks, and software composition analysis to identify vulnerable third party dependencies. If you bake these tools into your continuous deployment pipeline, this ensures every single code change meets your security requirements, and you validate your security posture many times a day. Not only that, but any security finding will be linked directly to an individual code change, which makes it easy to identify an owner, and to troubleshoot and resolve. This is particularly helpful in an enterprise context, where security fixes will be subject to service-level agreements.

Second, because each code change is so small, you can take a zero tolerance approach to test failures. If any security test fails, you simply stop your continuous deployment pipeline, and the change never makes it into production. Teams can quickly roll back the offending code, or roll forward with a fix and another deployment. This is a luxury you could never indulge in when deploying monthly or quarterly releases in an enterprise context, because engineering and security teams will be under immense pressure to get releases shipped in a predetermined release window.

And of course, continuous deployment eliminates the risk of introducing defects or security issues from integrating multiple changes at once. Small, individual changes are easy to review and reason about, so engineers can accurately evaluate the security impact of each one. This is true regardless of organizational size, but it’s particularly impactful at enterprise scale, where infrequent releases may include thousand of changes, from dozens or even hundreds of teams.

None of these security benefits are specific to enterprise software vendors. But they’re often overlooked when evaluating continuous deployment in an enterprise context. That’s a shame, because enterprise software vendors face greater security threats, integrate more security tools, and make more code changes. As a result, they benefit more from continuous deployment practices.

Compliance Frameworks

Beyond the safety and security measures you implement, enterprise software customers often expect you to produce a SOC 2 report, or ISO 27001 certification. This means you need to document your engineering and information security practices, and pass an annual audit. Your software delivery process is in scope for both SOC 2 and ISO 27001 — so when you first introduce continuous deployment, you will need to work with your compliance team to accommodate the changes.

Helpfully, enterprise software vendors will have their software development workflow documented in detail in a set of standard operating procedures. These documents cover all processes related to software delivery, and can act as a template for your transition to continuous deployment. For each manual deployment activity covered in your standard operating procedures, you will need to design, implement, and document an automated process that offers equivalent or better safety. This is often surprisingly easy, because manual processes involve a tradeoff between safety and usability. When you automate these, you can build additional safeguards into the system that would be too onerous to handle manually. And by automating the workflow, you also reduce the training requirements and cognitive load for your team.

When reviewing SOC 2 and ISO 27001 compliance, auditors will look at whether your development process implements adequate security controls, and whether the process is actually being followed. Here, continuous deployment helps in two specific ways. First, by automating every aspect of the deployment process, you reduce the risk of human error — or malicious activity by insiders. Second, if your deployment tooling has comprehensive logging, this creates an excellent audit trail for your software delivery. You can use this in future audits to prove your compliance controls are working.

One specific compliance control auditors will pay close attention to with continuous deployment is “segregation of duties”. In a nutshell, no single person should be able to cause damage to the system — either inadvertently or maliciously. The traditional enterprise way to address this is to have a change control board or QA team review and approve every change before it goes to production. This gets the job done, but it’s also incredibly heavy-handed.

A much simpler solution for segregation of duties is something you already do today: Pull request reviews. If you configure your source control system to require at least one approval on each pull request to trunk, engineers can no longer make production changes unilaterally. Continuous deployment ensures each deployment is linked to a single pull request, which gives auditors total clarity on who authored and who approved each change. You can also add automated testing to the pull request process, to block merges until changes pass all tests. And in highly regulated industries, you can automatically identify sensitive pull requests — such as changes to access control configuration — and add mandatory reviewers from your security or compliance teams. These measures will satisfy the segregation of duties requirement in most environments.1

Ultimately, a high degree of deployment automation is better from a compliance perspective, because it provides a consistent process, strict enforcement, and an excellent audit trail that you can use to pass future process audits. And continuous deployment gives compliance teams and auditors full visibility of each individual code change, along with associated test results, code reviews, and approvals.

Deployment vs Release

With the security and compliance concerns out of the way, there’s still the matter of shipping product features continuously. Enterprise customers in compliance-driven industries are not fond of frequent product changes, because each update may trigger internal change management work. This is especially true in life sciences, where GxP guidelines may require new product features to be captured in internal process documentation.

The notion here is that you can’t change enterprise software without letting customers know in advance, documenting the change, and giving everyone time to adjust. This is where feature toggles come in. Feature toggles allow you to control the visibility of new features until you’re ready to make them available. This in turn allows you to separate deployment from release: You can deploy your work on a new product feature to production in hundreds of small increments without making any part of it visible to end users. Product managers or other customer-facing functions can then decide when to release completed features.

The drawback with this approach is that you don’t get feedback on new features until someone enables them. And if you’re not getting continuous feedback, you erode the value of doing continuous deployment in the first place. So rather than toggling features globally, you may want to enable new features for individual customers, and for test accounts in production. With some tooling to manage this, you can launch a “pioneer” program to give progressive customers early access to new features. This is a fantastic opportunity to build closer relationships with your most engaged customers, and collect their feedback on features under development. At the same time, customers that operate hospitals or airports can hold off on adopting new functionality until it’s been thoroughly vetted by the early adopters.

It’s a good idea to establish an internal agreement on what changes can be made when. Even enterprise customers want critical bugs fixed immediately — and they’ll appreciate your ability to turn around these fixes in hours or minutes. No need to feature toggle these. You may also be able to continuously release minor bug fixes and incremental features, so long as they don’t impact existing workflows. This is especially true for features that are not visible to end users by default. That means you may only need to feature toggle major functional changes, or changes that cannot be controlled by your customers’ administrators.

Whatever your approach, you need align on it with your compliance team, and with representatives from customer-facing functions. Having a clear written agreement on what’s acceptable helps engineers make good deployment decisions, and creates a predictable software release experience for customers.

Next Steps

Continuous deployment allows you to ship enterprise software in a manner that’s safe and secure, aligned with common compliance frameworks, and doesn’t disrupt customer activities. In many ways, continuous deployment is more impactful in enterprise settings, because it solves problems that don’t exist in smaller companies: Large organizations make more code changes, integrate more security tools, and submit to greater scrutiny of their delivery processes.

If you’re introducing continuous deployment in an enterprise environment for the first time, expect some initial skepticism from compliance and release management teams, and from customers. You’ll have to earn their trust by engaging them in the process, and building out automated tools to directly address their needs for safety, compliance, and consistency.

As with any technical and cultural transformation, it helps to tackle this incrementally. I recommend starting with processes in your delivery workflow that involve tedious manual labor, or that are hotspots for compliance audit findings. If you automate these, you achieve two immediate goals: You free up time that you can reinvest in making further changes, and you get some runs on the board, to prove the model leads to better outcomes.

Once you’ve established this foundation of trust, you will have an easier time building out the tooling and processes for fully automated continuous deployment. Happily, getting over these initial objections is the hardest part — because you build up a track record of safe deployments really fast when you deploy to production 20 times a day.

Here, I’d like to share my views on building software products and software engineering organizations. Posts will be loosely themed around achieving worthwhile outcomes through incremental changes. Expect topics like continuous deployment, lean product engineering, remote work, professional development, and engineering culture.

In the spirit of incrementalism, I plan to start shipping, and learn along the way. So please feel free to share feedback or start a conversation in comments, on Twitter, or via email.

I’m happy you’re here! Now let’s get to it.

Tobias