In our previous blog post about the Istio service mesh, we provided an overview of Istio’s features and capabilities and why you may (and sometimes may not, at least not yet) want to use it as a service mesh in your Kubernetes clusters. In this post, we’ll dive a little deeper into how Istio can help improve the runtime security of the applications in a service mesh and where it fits in the broader picture of Kubernetes security controls and practices.

Trust No One

At its most hardened, Istio provides a large chunk of the functionality needed to support the ability to run microservices securely on zero-trust networks.

The zero-trust model is built on the possibility that networks and infrastructure can be infiltrated by malicious or misconfigured code or other bad actors. It assumes that, just because any given service is running on your Kubernetes cluster, it does not mean it should be there or that it can be trusted without question (Just ask Tesla). In a distributed set of interconnected microservices, new services can spring into existence literally overnight and other services can potentially start using them (or get called by them) without an overt configuration change. That makes the zero-trust philosophy of identify and verify, in conjunction with other application CI/CD and infrastructure best practices, a key component of the security management of applications running in a Kubernetes cluster.

For its part in supporting a zero-trust network, Istio offers a mix of additional layer 4-7 firewall controls, encryption over the wire, and advanced service and end-user authentication and authorization options for services in the mesh, along with the ability to capture detailed connection logging and metrics. As with most of Istio’s capabilities, these are all powered under the hood by the Envoy proxy running as a sidecar container beside each application instance (Kubernetes pod) in the service mesh.

mTLS: Trust but Verify

mTLS (Mutual Transport Layer Security) is a fundamental piece of the Istio security toolset. Not only does it provide encryption over the wire, it also enables service-to-service authentication and authorization in a service mesh.

How It Works: Each service in the mesh has its own TLS certificate, which is issued and managed by the Istio Citadel service. In a given network connection, the Envoy proxy of the client service will present its certificate to the server, while verifying the validity of the server’s TLS certificate. The Envoy proxy of the target service will verify the client certificate, and it can also use the identity of the client to determine if that service is allowed to connect at all, and if so, what it is authorized to do, based on the Istio service RBAC (Role-Based Access Control) configuration and the service mesh and policy configuration. The TLS encryption of the connection also means that even applications which do not natively support TLS encryption will not send plain-text traffic over the network.

Istio Authorization RBAC acts very much like an extension of native Kubernetes RBAC. ServiceRoles can be defined that create per-endpoint permissions for services in the mesh. These can be bound to authenticated entities like Kubernetes service accounts or external users authenticated with JWT tokens to permit service access based on identity. The Istio RBAC evaluation is done in the proxy sidecar, making it very low-latency.

All of the above is completely transparent for most applications, requiring no configuration change to the deployments.

Action Needed: Global mTLS is not on by default. You can set it globally at Istio install time with the global.mtls.enabled=true setting, or turn it on per namespace or globally later.

Istio RBAC needs to be enabled and configured, starting with a ClusterRbacConfig resource.

Potential Issues:

  • Applications which have their own HTTPS built-in with no option for HTTP (usually for using client certificates of their own for authentication) will not benefit from Istio’s mTLS, which doesn’t support tunneling TCP, and they will need additional Istio configurations to bypass mTLS enforcement.
  • In a cluster where only a subset of services or namespaces have mTLS enforcement enabled, care must be taken to isolate resources in the cluster that are not part of the service mesh by using Kubernetes Network Policies, where supported.

Policy Checks: Fine-Grained Authentication and Authorization Controls

Mixer, Istio’s policy control service, enables a number of ways to add access control to applications in an Istio service mesh. Not only does it ship with a number of adapters out of the box, its pluggable adapter model allows users to deploy and use their own verification mechanisms if needed.

While we’ve already seen that using mTLS enforcement and Istio RBAC in a mesh handles the service-to-service authentication and authorization, incoming connections from outside the cluster are generally not going to have their own valid Istio mTLS certificates. Mixer already includes support for JSON Web Tokens (JWT) for authenticating end-users or external upstream services, or you can create policy checks for different authentication methods using a custom Mixer adapter.

Once the connection has been authenticated, policy checks offer countless options for deciding whether to allow access to the requested endpoint. Istio offers a large number of connection attributes that can be used in the checks, depending on the port protocol. The options are richest for non-TLS HTTP services, where the URI path, request header content, query parameters, and other attributes can be evaluated. Istio RBAC Authorization can also be used, although it may sometimes be less useful for connections originating outside the service mesh because RBAC evaluation is based on static rules.

How It Works: When policy checks are enabled for a mesh, the target Envoy proxy will connect to the Mixer policy service for each incoming connection to confirm whether to allow or refuse the request. Mixer determines which policies apply to the service and evaluates the attributes for the requested connection against them.

Action Needed: Policy enforcement, required for Mixer checks to work, is also not enabled by default. It can be enabled at install time with the global.disablePolicyChecks=false flag, or it can be turned on later.

Potential Issues:

  • Enabling policy checks adds some latency to connections, because the target Envoy proxy must connect to Istio’s Mixer Policy service to confirm authorization for each request.
  • When using custom or third-party adapters with Mixer, care must be made to ensure the quality and reliability of the code and the deployment to prevent latency and downtime if the backing service becomes unavailable and to ensure authentication and authorization results are correct.

Egress Traffic Control

Kubernetes has helped accelerate the speed with which developers can deploy new applications and iterate releases for existing applications in large part by decreasing friction around infrastructure resource allocation and by standardizing deployment configurations. That speed has not reduced the need to monitor the behavior of the application, of which network egress is an important part. It becomes increasingly important to know and control what third-party services are being accessed, for auditing purposes, for data security, and to limit the potential for damage if any malicious code somehow finds its way into a running container in the service deployment.

Istio provides the ability to block traffic to all endpoints outside the service mesh by default, requiring any valid external services to be explicitly on an allow list.

How It Works: If the mesh’s outbound service policy is set to REGISTRY_ONLY, when the Envoy proxy for an application intercepts an outgoing connection to a destination outside the service mesh, it checks to see if the destination has been configured with a ServiceEntry. If so, it completes the external connection. If not, it returns an error to the source application or drops the connection, depending on the protocol.

Further control and monitoring of egress traffic can be facilitated by deploying an Istio egress gateway and routing outgoing traffic through that. When combined with Kubernetes Network Policies that only allow Istio’s egress gateway deployment to make connections outside the cluster, these measures can also help prevent any rogue container or process in the Kubernetes cluster from bypassing the Envoy proxy and making direct outgoing connections on its own.

Action Needed: Egress traffic is open (unrestricted) by default. It can be locked down at install time with the global.outboundTrafficPolicy.mode=REGISTRY_ONLY flag, or it can be turned on later.

Potential Issues:

  • Moving from an open outbound policy to an allow list-only one can be very difficult, as so many in-house and third-party applications rely on SaaS services. Even when you think you know what external sites your containers are accessing, along come a few HTTP redirects, and your egress connections still fail. HTTP 301/302 redirects are returned to the client, which then has to make a new connection to the new location. One HTTP GET can easily become multiple layers of redirects, each of which needs to be on an allow list.

    $ curl -sL -D - -o /dev/null http://www.docker.io/
    HTTP/1.1 301 Moved Permanently
    Content-length: 0
    Location: https://www.docker.io/

    HTTP/1.1 301 Moved Permanently
    Content-length: 0
    Location: https://www.docker.com/

    HTTP/2 200
    content-type: text/html; charset=UTF-8
    content-length: 86100

    [...]
  • Istio does not log failed egress connections consistently, varying with the Istio version, the protocol, and cluster configurations. Discovering the exact targets of outbound connections can be difficult.

  • Forcing all egress traffic through an egress gateway by default is borderline impossible. Istio does not provide a global gateway configuration configuration, and the VirtualService resources used to direct egress traffic to an egress gateway have limited wildcard handling for destination addresses, mainly due to limitations in the Envoy proxy.

A Swiss-Army Knife, Not a Craftsman Toolbox

While Istio certainly provides a number of controls for application network security, it is not a comprehensive solution. It is best used in combination with lower-level infrastructure network controls and general Kubernetes security best practices.

One very important gap to watch out for is the lack of support for any protocol other than TCP, because that is the only layer-4 protocol that the Envoy proxy currently supports. This means if you need to monitor or control UDP or SCTP traffic in your Kubernetes service mesh, you will need to use Kubernetes Network Policies, or, if your Kubernetes service provider does not support Network Policies, you will have to see if they offer alternative network controls. When available and configured, Network Policies make a very nice complement and enhancement to Istio’s own controls.

And, of course, you have to protect the security of the Istio resources in your cluster to ensure Istio’s protections cannot mistakenly or maliciously be overridden or disabled. Because Istio is configured completely with Kubernetes standard resource definitions, like ConfigMaps and Secrets, along with Istio’s own Kubernetes Custom Resource Definitions (CRDs), any person or service account with cluster-level RBAC admin privileges could modify the configuration. Use Kubernetes namespaces to group workloads logically, be sure to restrict RBAC privileges with the principle of least privilege, and deploy and harden Istio following recommended best security practices.