50 Ways to Break Production - Why Cloud Security Remediation Is Harder Than It Looks

Intro

Cloud security today is mostly built around three ideas: visibility, prioritization, and remediation.

Most major cloud security platforms, whether CSPM or CNAPP, focus heavily on the first two. They identify insecure resources and prioritize findings, this is where prevention and visibility diverge: CSPM and CNAPP are built to show a misconfiguration after it's deployed, not stop it from being created in the first place. The actual fixing process is handed over to the engineering, DevOps, or platform teams.

And for good reason.

Anyone who has tried to remediate cloud misconfigurations in a real production environment knows that remediation is not a simple checkbox. A seemingly minor configuration update can disconnect services, revoke permissions, block runtime access, or break production in completely unexpected ways.

This article reviews common remediation challenges, why they are harder than they look, and how organizations can approach them more safely.

Resource Configuration Updates That Will Definitely Break Production

Let’s start with a simple example: an Azure Storage Account whose network access is enabled from all networks.

This example is not unique to Storage Accounts. The same pattern applies to many serverless resources across Azure, AWS, and GCP.

Imagine we have a functioning production system that depends on this Storage Account. The Storage Account is currently reachable from all networks. From a security perspective, this is clearly a problem. A typical recommendation would be:

Disable public network access and restrict access to approved networks only.

At first glance, this sounds straightforward. In reality, there is only a very small chance that everything is already correctly configured behind the scenes: that a private endpoint exists, that every dependent workload communicates through it, that DNS is configured correctly, and that the only thing left to do is disable public access.

More likely, changing this setting will break the runtime environment. So what do organizations do?

Option 1: Give Up

This is more common than most people think. The finding stays open. The team accepts the risk. Everyone agrees that the configuration is insecure, but no one wants to touch it.

In many organizations, this is the default outcome of the ‘remediation’ approach: the issue is visible, prioritized, discussed, and then left unresolved.

Option 2: Work Very Hard

The safer path is to treat remediation as an architecture change, not as a configuration toggle. That requires more effort, but in many cases, there is no shortcut.

First, we need to understand what secure networking options are actually available.

When restricting network access to Azure PaaS resources, the options are:

IP or CIDR Allowlisting

This is useful for non-production environments such as dev, sandbox, or staging. It can also work for third-party integrations that provide a static IP address or a narrow CIDR range.

For example, an organization may allowlist specific IT workstation IPs to support testing and debugging while still maintaining a basic network perimeter. However, IP allowlisting is fragile. It only works safely when the source IPs are static and well understood.

It has two main drawbacks:

Operational: It is difficult to maintain. If applied to the wrong architecture, it can require frequent updates and become a source of outages or access issues.

Security: While it is far better than exposing services to the public internet, IP allowlisting still relies on network-level trust and may remain vulnerable to man-in-the-middle attackers or threat actors capable of spoofing source IPs.

Service Endpoints

Service Endpoints, also known as VNet Endpoints, are usually a good fit for internal cloud workloads.

They are free, relatively easy to implement, and allow traffic to Azure services to stay within the Azure backbone. For many internal workloads, this can be a practical and secure option.

Private Endpoints

Private Endpoints are generally recommended for highly sensitive workloads, third-party integrations, and direct connectivity between PaaS services and on-premises networks through Private Link.

They provide stronger network isolation, but they also require more careful planning, especially around DNS, routing, and cross-environment access.

What About CI/CD pipelines?

Now let’s take a complex example: a GitHub Actions workflow that needs to access the Storage Account.

At first, none of the options above seem to work cleanly. In many cases, however, a secure design is still possible. You could use self-hosted runners with static outbound IPs and allowlist only those addresses. You could connect the runner environment to a VNet using a VPN or private connectivity solution. From there, you could use a Service Endpoint or Private Endpoint.

The important point is this:

The secure solution usually exists, but it is rarely a one-line remediation. It requires design, testing, and operational ownership.

Additional tips

Review access logs before making changes

The most important step is understanding who and what currently accesses the resource.

For Storage Accounts, this means enabling data access logs through diagnostic settings and analyzing actual access patterns. This analysis needs to be done carefully. If you plan to use Service Endpoints, you need to understand which VNets access the resource today.

If you plan to use IP allowlisting, you need to confirm that the observed source IPs are static. Too often, teams look at access logs, collect all the IPs they see, allowlist them, and later discover that those IPs were temporary or dynamically assigned.

Logs tell you what is happening now, but they do not automatically tell you what is safe to depend on.

Always model the desired architecture first

Before changing anything, create a new Storage Account configured according to the desired architecture and try to make the required communication work there first.

Can the application connect? Can the CI/CD pipeline access it? Can operational teams debug issues? Can third-party services still integrate?

This gives the team a safe environment to validate the future secure architecture before applying it to production.

Resources That Cannot Be Changed After Creation

Let’s take encryption at rest for RDS (Relational Database Service) as an example. The setting is a simple boolean field, but unfortunately it is immutable after creation. Even when no architectural redesign is required, remediation can involve a long operational process: exporting or migrating data, provisioning a new resource, validating the new setup, and potentially accepting downtime unless significant engineering effort is invested in minimizing migration impact. All of this is required because the setting cannot be changed in place.

The same pattern applies to Azure Key Vault access policies, Storage Account geo-redundancy, Disk Encryption Set double encryption, AWS SageMaker network access, and EBS encryption at rest. All fall into the same category of settings that are cheap to get right at creation and expensive, if not impossible, to fix later.

What all of these cases have in common is that remediation creates a major operational burden, while prevention often only requires setting the correct configuration at creation time. In other words: prevent it before deployment, and there's nothing left to remediate.

Organizational and Psychological Barriers

There’s another major problem with the remediation approach: an organizational and psychological barrier.

Why would an organization pull its operations teams away from planned work to revisit infrastructure that is already running? From a business perspective, there is always pressure to keep moving forward and continue building. This creates an incentive for team leads to argue about findings, downplay the security risk, or decide that it simply isn’t important enough to prioritize. Organizations operating reactively are constantly forced to choose between “spending” resources on remediation and architectural changes, or continuing to move forward with development.

In this model, it becomes very difficult to invest in proper architecture or address security issues that are important but not urgent - the kind of issues that matter, but don’t come with a clear “we’re going to get breached tomorrow if we don’t fix this” deadline. Organizationally, it usually makes much more sense to build things securely from the start. Once a project is already built and running, it becomes much harder to get the time, attention, and resources needed to go back and fix it.

Auto-Remediation and IaC

One of the most appealing paths organizations might take is automated remediation. But not all that glitters is gold.

Beyond providing a faster, automated way to break production, auto-remediation introduces another fundamental challenge: ownership.

Who owns the resource?  Usually it is the operational team.

That team typically manages the resource through Terraform, CloudFormation, or another Infrastructure-as-Code (IaC) framework. So what happens the next time the operator modifies the desired state and runs “terraform apply”? The remediation is reverted.

IaC is the source of truth for the resource, meaning its state is ultimately controlled by the IaC configuration. Any change made directly to the resource by an auto-remediation process can therefore be overwritten the next time the IaC is applied. This is the sharp edge of infrastructure as code security: a fix that isn't reflected in code isn't really a fix, it's a countdown until the next apply.

This is why organizations often struggle to adopt even simple auto-remediation actions, even in the rare cases where remediation is safe to production.

Option 3: Prevention - The Path of Least Friction

All roads lead to Rome Prevention. What if we just prevent misconfigurations before they reach the runtime environment?

Why choose the hard path?

Once an insecure configuration becomes part of a working system, it becomes much harder to remove.

Security teams shouldn’t only ask: How do we fix this?

They should also ask: How did this reach production in the first place?

And, more importantly: How do we make sure the next insecure deployment is blocked before it becomes a dependency?

Surprisingly, prevention avoids many of the problems that make remediation difficult.

First, there is the organizational process. Instead of security teams having to constantly chase operations teams to fix issues, security and operations teams can define organizational policies together, and enforced them automatically.

Instead of spending resources fixing existing infrastructure, the organization can focus its effort on moving forward, while policies ensure that new deployments do not introduce security or architectural misconfigurations.

What about production risk? When prevention is implemented correctly, it avoids the operational risk introduced by remediation. And what about Terraform, ownership, and IaC? Preventive policies behave like native validation errors returned by the cloud provider, a failure mode that IaC workflows are already designed to handle. The deployment fails, the error is surfaced, and the operator can immediately see what needs to change.

Good prevention does not change ownership within the organization. It gives operational teams the right guardrails to work independently within the boundaries defined by the organization. This is exactly what a cloud security enforcement platform does - enforcing those guardrails, at the moment a resource is created, without taking ownership away from the team that owns it.