Why a move to containers means a giant leap forward for incident response

Container technology is fundamentally changing the way incident response (IR) is handled within the enterprise, and it is putting agile organizations back in a position of strength against their attackers. Microservices and containers comprise an infrastructure that can be leveraged as a security orchestrator and responder, which allows for radical improvements in both the scale and speed of threat detection, response, and prevention.

IR in a traditional environment

Today’s systems have become too distributed, integrate too many programs, and present too many attack surfaces for security analysts to thwart attacks effectively. To discover where an attack originated in these systems — or to understand what it did on a particular endpoint system, or how it moved laterally, or how it communicated in and out to receive updates — you have to collect diverse data from across a fragmented IT infrastructure that includes different types of applications, versions, and users, and even custom data from systems, network, and memory.

From a security perspective, gathering this information typically means leveraging a large number of point solutions or tools that cross internal boundaries between perimeter-based solutions and network-based solutions, in addition to offline solutions, like analytical log-based solutions. It can also mean crossing boundaries between areas governed by different sets of regulatory compliance requirements, such as human resources, or legal, or finance and IT.

So in order to drill down to the root cause of a disruption, you end up having to create multiple matrices of problems you have to analyze and resolve using as many as 50 or 60 different tools.

These tools have come a long way since the early days when IR and forensics were primarily manual processes that were implemented by on-site specialists. Cybersecurity analysts have learned to identify patterns that reveal attack techniques, and they have developed solutions for automating data collection. However, traditional monitoring and detection tools are not effective at web scale, and do not lend themselves to orchestration for security automation.

Containers and microservices architectures change IR practices

The adoption of solutions such as Docker, Kubernetes and microservices architectures allows us to address this problem in a new and elemental way. It enables us to build security, monitoring, and IR using the same building blocks as the application itself, so that security is seamlessly integrated into the architecture. Instead of overlaying security solutions onto a pre-existing architecture, containerization allows us to collect data from all containers across the environment, and detect and automatically respond to anomalies and threats.

Containers make this possible because they break applications down into their basic functions, and they standardize the assignment of specific resources within each container. This allows us, in turn, to standardize how data such as system calls, network traffic, application-level data and Docker events are collected from container environments. By collecting data from the entire software stack, we can correlate signals with greater efficiency, because a well-defined behavioral modeling system can reduce the noise from data sources that include operating systems and users’ actions.

From the perspective of the container platform, each environment is an immutable replica of the other environments that are running the same platform. This allows us to subject the data to highly sophisticated detection models that establish a baseline for container behavior, trigger alerts on anomalies, and locate them in the affected services, according to each business’ use cases. The advantage here is that instead of simply being able to identify a single infected host system, the broader context is available in order to identify the business application or system that was compromised.

IR in a microservices environment

Container environments challenge the investigative aspects of incident response, which rely heavily on the preservation of stateful data in order to study an attack and learn the attacker’s techniques and craft. Containers are ephemeral: they are only generating data if they are running. As soon as they stop running, they stop producing the data that would allow you to watch the attack directly. More importantly, you lose the stateful data that is required to determine how an attacker infiltrated the system.

Without container persistency, you can only get a sense of the attack from its symptoms or effects, but container technology still allows you to respond in a couple of ways once the system triggers on an event, and these options have their advantages in terms of increasing security. For example, if you set the IR system to shut down an infected container and restart it from scratch, you effectively cause an attacker to suffer from short-term memory loss, since they lose any persistency from the container where they launched their attack. This allows for some flexibility in the IR workflow and gives the organization a few options in terms of how to respond to an attack.

Quick detection also forces attackers to move hastily in a constantly changing environment, where they risk further detection by exposing their tools and techniques. It also limits the scope of an attack, as IR in a container environment enables us to log events at a system level.

When detection mechanisms are built into the instrumentation, though, you are not flying blind. In a containerized environment, you can put gates in place that automatically trigger the system to collect more in-depth data around the time of an event, before and after. This raw event data allows you to see which filter triggered, and compare that behavior against the larger cluster to see exactly how an attacker is moving. Having identified the attack, you can bookmark the event and create a policy around it. Now, you’re in a position to implement preventative measures that harden the infrastructure.

This process effectively creates automatic enforcement, with substantial increases in efficiency and a much lower false positive rate than a traditional IR solution.


By deploying a container security solution driven by instrumentation combined with machine learning and analytics, companies can reduce their security teams’ IR workload by more than 10 times, and ultimately reduce the number of security tools from over 50 to down to as few as 10.

StackRox is a great example of a comprehensive container security solution that enables security investigations by surfacing indicators of compromise (IOCs) that give users a sense of attack techniques. It also offers fully-automated response to threats based on user-defined policies.

Learn more about how StackRox detects, prevents, and responds to changing threats.