NEW REPORT: State of Container and Kubernetes Security 2019 DOWNLOAD NOW
{ .link_text }}

Docker Hub Breach: What to Do Now

What happened?

In an email to customers, Kent Lamb, Director of Docker Support, wrote “During a brief period of unauthorized access to a Docker Hub database, sensitive data from approximately 190,000 accounts may have been exposed (less than 5% of Hub users). Data includes usernames and hashed passwords for a small percentage of these users, as well as Github and Bitbucket tokens for Docker autobuilds.”

As a result of this breach, it’s possible that images in your Docker Hub repository may have been tampered with or overwritten. To mitigate this possibility, you should take immediate action. You can follow the recommendations below for any deployment and also utilize StackRox to take steps for additional protection.

Hardening Docker: A Security Toolkit

Download our practical guide on hardening Docker containers and hosts for a more secure container environment

Download Now

Rotate your secrets

Despite the passwords being hashed, you should change your passwords. Docker has already revoked the Github access keys and tokens that may have been used.

For Docker Hub user accounts, rotating secrets is a fairly straightforward process:

  1. Browse to https://hub.docker.com/ and login to your account.
  2. Open the drop down under your username and select “Account Settings.”
  3. Scroll down to Password and update!

If you use Docker Hub with Kubernetes environments, you’ll also need to roll your ImagePullSecrets, which can be more involved. The first step is to identify all of the ImagePullSecrets in your clusters, which can be done by running the following kubectl commands:

kubectl get secret --field-selector=type=kubernetes.io/dockerconfigjson --all-namespaces
kubectl get secret --field-selector=type=kubernetes.io/dockercfg --all-namespaces

Ideally the secrets are named in such a way that you can identify to which registry they correspond. If not, you can parse the secret as a last resort to determine the corresponding registry. The following kubectl command will print out all the registries referenced in your image pull secrets:

kubectl get secret --field-selector=type=kubernetes.io/dockerconfigjson --all-namespaces -o json | jq -r '.items[] | {namespace: .metadata.namespace, name: .metadata.name, registries: .data[".dockerconfigjson"] | @base64d | fromjson.auths | keys[]}'

Sample output:

{
"namespace": "default",
"name": "gcr",
"registries": "https://us.gcr.io"
}
{
"namespace": "default",
"name": "stackrox",
"registries": "https://docker.io"
}

Now delete the image pull secrets referencing docker.io and recreate them using kubectl (don’t forget to put them in the proper namespaces!).

Audit your images

Because Docker didn’t provide a specific timeline for this breach, no one knows how long ago the unauthorized access occurred. As with most breaches, the perpetrators may have had access to compromised resources significantly longer than just last week. To be safe, you should verify recently pushed images going back over the past several weeks. Doing this audit can be difficult as not every registry will let you filter the data by image age.

To help pull manifests and check the age of your images, you can use an open source tool such as the registry CLI available here (we have alternative suggestions for StackRox customers, later in this post). Here’s an example output:

repo=registry-1.docker.io/library/nginx
tag=latest
v1digest=$(reg manifest "${repo}:${tag}" | jq -r .config.digest)
reg layer "${repo}@${v1digest}" | jq '{created: .created, history: .history}'

Sample output:

{
"created": "2017-03-27T19:48:34.817993292Z",
"history": [
{
"created": "2017-03-21T18:28:51.055495122Z",
"created_by": "/bin/sh -c #(nop) ADD file:4eedf861fb567fffb2694b65ebdd58d5e371a2c28c3863f363f333cb34e5eb7b in / "
},
{
"created": "2017-03-21T18:29:05.091744235Z",
"created_by": "/bin/sh -c #(nop) CMD [\"/bin/bash\"]",
"empty_layer": true
},
...
}

Inspect your environment for unexpected activity

Attacks on the build pipeline can have serious downstream effects on what is currently running inside your infrastructure. Tainted images can be difficult to detect, and the containers launched from them may even run as expected, except with a malicious process in the background. Backdoored images with cryptominers have also been discovered on Docker Hub before.

Unexpected changes in images will have an effect on application behavior, making runtime detection and application baselining critical. Characterizing the behaviors of individual Kubernetes deployments will highlight deviations in network connectivity, file access, and process executions. These deviations are all indicators that malicious activity is taking place within a container. You need the ability to quickly inspect runtime activity within your containers to verify they are running only expected processes.

In these breach scenarios, tooling that enables an effective triage process is essential. Being able to quickly identify what resources are affected, determine a solution, and execute efficiently across your entire container environment are key enablers to reducing risks to your business.

Leverage StackRox to streamline incident response

StackRox provides visibility and security globally across your Kubernetes infrastructure. Key features in the StackRox Kubernetes Security Platform allow you to rapidly address risks that arise from breaches like this Docker Hub compromise.

Rotating Secrets

StackRox collects and indexes Kubernetes secret metadata to allow for quick accounting of the different secrets being managed. In this case, a search for ImagePullSecrets that reference docker.io and are more than 1 day old yields a queue of passwords that need to be updated and the environments in which they are located.

Auditing Images

StackRox enables you to quickly find Dockerfiles across your infrastructure. The simple query seen in the following screenshot returns all of the images that are using docker.io that have been built in the last 30 days. This capability allows you to inspect the content of those images and determine whether they contain any suspicious layers.

Inspecting deployment behavior

StackRox’s runtime detection provides visibility into all executed processes and will alert on known malicious processes. In conjunction with image auditing, runtime detection lets users quickly analyze the deployments that may have been compromised and verify no process irregularities are present.

Summary

Following events like this Docker Hub breach, you need tools and processes that help you rapidly identify where you’re vulnerable to the security lapse. Security platforms purpose-built for containers and Kubernetes can make these tasks easier. Here at StackRox, we combine configuration checks, runtime detection, powerful search features, and other capabilities to make it easier for you to triage issues and identify problem spots throughout your environment. We’d love to hear feedback on additional out-of-the-box policies that would make it easier to understand and characterize what’s running in your environment. Want to see more? Request a demo. Have ideas on new policies we should create? Drop us a note. We’re always looking to improve our product for container and Kubernetes security.


Categories:Tags: