Enterprise-Grade Continuous Deployment
Improve security, compliance, and release management outcomes with continuous deployment in the enterprise.
I started out building software for small and medium-sized business customers. When I moved into enterprise software development over the last few years, I was taken aback by the resistance to modern engineering practices, and to continuous deployment in particular. It’s taken as gospel that continuous deployment won’t work for enterprise-grade software, or for compliance-heavy industries, like healthcare, aviation, or government.
I disagree. You can absolutely do continuous deployment in enterprise environments. And just like in smaller businesses, it will lead to better outcomes for you and your customers. In my most recent role, my teams built software for some of the largest organizations in the world. Collectively, these companies used our products to support millions of employees globally — many of them working in compliance-driven fields. Despite this, my teams deployed to production more than 2,500 times in 2021, with dozens of deployments on busy days. We delivered a steady cadence of product improvements, with consistently high quality and no disruption to service. And we were able to turn around defect fixes for customers in as little as 25 minutes.
So why the angst about using continuous deployment in enterprise environments? The objections I’ve heard generally focus on three things enterprise software customers expect from their vendors:
You need to meet stringent safety and security requirements
You need to communicate upcoming product changes well in advance, to avoid surprises
I’m here to tell you that continuous deployment is not only compatible with these requirements, but it helps you achieve them. Let’s take a look at how.
Safety & Security
Large companies have strict requirements for safety and security. And justifiably so: The legal, financial, and reputational impacts of security incidents can be huge. When these companies purchase software, they make sure their vendors have a robust and multi-layered security strategy in place, to protect against internal and external threats. Your software delivery process has a key part to play here, and this is where continuous deployment helps in several specific ways.
First, you can integrate many security tools directly into your development workflow. This includes tools for static security testing, application configuration checks, and software composition analysis to identify vulnerable third party dependencies. If you bake these tools into your continuous deployment pipeline, this ensures every single code change meets your security requirements, and you validate your security posture many times a day. Not only that, but any security finding will be linked directly to an individual code change, which makes it easy to identify an owner, and to troubleshoot and resolve. This is particularly helpful in an enterprise context, where security fixes will be subject to service-level agreements.
Second, because each code change is so small, you can take a zero tolerance approach to test failures. If any security test fails, you simply stop your continuous deployment pipeline, and the change never makes it into production. Teams can quickly roll back the offending code, or roll forward with a fix and another deployment. This is a luxury you could never indulge in when deploying monthly or quarterly releases in an enterprise context, because engineering and security teams will be under immense pressure to get releases shipped in a predetermined release window.
And of course, continuous deployment eliminates the risk of introducing defects or security issues from integrating multiple changes at once. Small, individual changes are easy to review and reason about, so engineers can accurately evaluate the security impact of each one. This is true regardless of organizational size, but it’s particularly impactful at enterprise scale, where infrequent releases may include thousand of changes, from dozens or even hundreds of teams.
None of these security benefits are specific to enterprise software vendors. But they’re often overlooked when evaluating continuous deployment in an enterprise context. That’s a shame, because enterprise software vendors face greater security threats, integrate more security tools, and make more code changes. As a result, they benefit more from continuous deployment practices.
Compliance Frameworks
Beyond the safety and security measures you implement, enterprise software customers often expect you to produce a SOC 2 report, or ISO 27001 certification. This means you need to document your engineering and information security practices, and pass an annual audit. Your software delivery process is in scope for both SOC 2 and ISO 27001 — so when you first introduce continuous deployment, you will need to work with your compliance team to accommodate the changes.
Helpfully, enterprise software vendors will have their software development workflow documented in detail in a set of standard operating procedures. These documents cover all processes related to software delivery, and can act as a template for your transition to continuous deployment. For each manual deployment activity covered in your standard operating procedures, you will need to design, implement, and document an automated process that offers equivalent or better safety. This is often surprisingly easy, because manual processes involve a tradeoff between safety and usability. When you automate these, you can build additional safeguards into the system that would be too onerous to handle manually. And by automating the workflow, you also reduce the training requirements and cognitive load for your team.
When reviewing SOC 2 and ISO 27001 compliance, auditors will look at whether your development process implements adequate security controls, and whether the process is actually being followed. Here, continuous deployment helps in two specific ways. First, by automating every aspect of the deployment process, you reduce the risk of human error — or malicious activity by insiders. Second, if your deployment tooling has comprehensive logging, this creates an excellent audit trail for your software delivery. You can use this in future audits to prove your compliance controls are working.
One specific compliance control auditors will pay close attention to with continuous deployment is “segregation of duties”. In a nutshell, no single person should be able to cause damage to the system — either inadvertently or maliciously. The traditional enterprise way to address this is to have a change control board or QA team review and approve every change before it goes to production. This gets the job done, but it’s also incredibly heavy-handed.
A much simpler solution for segregation of duties is something you already do today: Pull request reviews. If you configure your source control system to require at least one approval on each pull request to trunk, engineers can no longer make production changes unilaterally. Continuous deployment ensures each deployment is linked to a single pull request, which gives auditors total clarity on who authored and who approved each change. You can also add automated testing to the pull request process, to block merges until changes pass all tests. And in highly regulated industries, you can automatically identify sensitive pull requests — such as changes to access control configuration — and add mandatory reviewers from your security or compliance teams. These measures will satisfy the segregation of duties requirement in most environments.1
Ultimately, a high degree of deployment automation is better from a compliance perspective, because it provides a consistent process, strict enforcement, and an excellent audit trail that you can use to pass future process audits. And continuous deployment gives compliance teams and auditors full visibility of each individual code change, along with associated test results, code reviews, and approvals.
Deployment vs Release
With the security and compliance concerns out of the way, there’s still the matter of shipping product features continuously. Enterprise customers in compliance-driven industries are not fond of frequent product changes, because each update may trigger internal change management work. This is especially true in life sciences, where GxP guidelines may require new product features to be captured in internal process documentation.
The notion here is that you can’t change enterprise software without letting customers know in advance, documenting the change, and giving everyone time to adjust. This is where feature toggles come in. Feature toggles allow you to control the visibility of new features until you’re ready to make them available. This in turn allows you to separate deployment from release: You can deploy your work on a new product feature to production in hundreds of small increments without making any part of it visible to end users. Product managers or other customer-facing functions can then decide when to release completed features.
The drawback with this approach is that you don’t get feedback on new features until someone enables them. And if you’re not getting continuous feedback, you erode the value of doing continuous deployment in the first place. So rather than toggling features globally, you may want to enable new features for individual customers, and for test accounts in production. With some tooling to manage this, you can launch a “pioneer” program to give progressive customers early access to new features. This is a fantastic opportunity to build closer relationships with your most engaged customers, and collect their feedback on features under development. At the same time, customers that operate hospitals or airports can hold off on adopting new functionality until it’s been thoroughly vetted by the early adopters.
It’s a good idea to establish an internal agreement on what changes can be made when. Even enterprise customers want critical bugs fixed immediately — and they’ll appreciate your ability to turn around these fixes in hours or minutes. No need to feature toggle these. You may also be able to continuously release minor bug fixes and incremental features, so long as they don’t impact existing workflows. This is especially true for features that are not visible to end users by default. That means you may only need to feature toggle major functional changes, or changes that cannot be controlled by your customers’ administrators.
Whatever your approach, you need align on it with your compliance team, and with representatives from customer-facing functions. Having a clear written agreement on what’s acceptable helps engineers make good deployment decisions, and creates a predictable software release experience for customers.
Next Steps
Continuous deployment allows you to ship enterprise software in a manner that’s safe and secure, aligned with common compliance frameworks, and doesn’t disrupt customer activities. In many ways, continuous deployment is more impactful in enterprise settings, because it solves problems that don’t exist in smaller companies: Large organizations make more code changes, integrate more security tools, and submit to greater scrutiny of their delivery processes.
If you’re introducing continuous deployment in an enterprise environment for the first time, expect some initial skepticism from compliance and release management teams, and from customers. You’ll have to earn their trust by engaging them in the process, and building out automated tools to directly address their needs for safety, compliance, and consistency.
As with any technical and cultural transformation, it helps to tackle this incrementally. I recommend starting with processes in your delivery workflow that involve tedious manual labor, or that are hotspots for compliance audit findings. If you automate these, you achieve two immediate goals: You free up time that you can reinvest in making further changes, and you get some runs on the board, to prove the model leads to better outcomes.
Once you’ve established this foundation of trust, you will have an easier time building out the tooling and processes for fully automated continuous deployment. Happily, getting over these initial objections is the hardest part — because you build up a track record of safe deployments really fast when you deploy to production 20 times a day.
The Cybersecurity & Infrastructure Security Agency agrees. In their Cloud Security Technical Reference Architecture, the agency suggests code reviews “by another authorized team member” as an alternative to traditional segregation of duties controls.