Integrating Charmed Kubernetes with AWS through OIDC

Canonical’s Charmed Kubernetes is a collection of open-source Kubernetes software program bundled with many addons, together with CNIs, CSIs, monitoring instruments and cloud integrations. As Juju helps AWS, Charmed Kubernetes will be deployed and run on AWS seamlessly. In reality, lots of our prospects run their Charmed Kubernetes on AWS for manufacturing workload. In such deployment, it is extremely pure for a pod to entry AWS sources. Whereas it’s simple to put your IAM entry key and secret key within the pod or a ConfigMap to get entry to AWS providers, it causes safety points too. For instance, these keys will be uncovered to others by merely retrieving the content material of pod configuration or ConfigMap knowledge. Furthermore, it’s common to rotate static keys periodically to cut back the affect of key loss; Even when the keys are stolen, the thief can solely use these keys for a brief time period, not without end. Nonetheless, rotating these keys is one other operation process which provides as much as on a regular basis’s work.

AWS gives STS (safety token service) that enables a service or a machine to amass short-lived tokens to entry AWS sources. This characteristic is obtainable on AWS managed Kubernetes service, EKS, through the usage of OIDC supplier in IAM. To set this up, the consumer must allow OIDC endpoint on EKS, then create an OIDC supplier in IAM pointing to that OIDC endpoint. This fashion IAM trusts tokens coming from this OIDC supplier and therefore points STS tokens in return. Element configuration will be discovered on this AWS doc. Below the hood, k8s pods depend on the EKS net identification webhook to amass a STS token. Fortuitously, AWS has open-sourced this webhook so customers can set it up in any Kubernetes cluster, regardless of operating in AWS or on-prem, to combine with AWS through OIDC.

OIDC in Kubernetes

As of right this moment’s newest model 1.24, Kubernetes has built-in assist for OIDC, which is documented in KEP 1393. In accordance with the official Kubernetes doc, OIDC is supplied by a characteristic referred to as Service Account Issuer Discovery. This characteristic turned steady in Kubernetes model 1.21, and within the present model it’s enabled when the characteristic Service Account Token Quantity Projection is enabled. It gives two OIDC-compliant endpoints, an OpenID Supplier Configuration doc at https://api_server:port/.well-known/openid-configuration and the related JSON Net Key Set (JWKS) at https://api_server:port/openid/v1/jwks. The JWKS doc incorporates public keys {that a} relying celebration can use to validate the Kubernetes service account tokens. Relying events first question for the OpenID Supplier Configuration doc, then use the worth of jwks_uri discipline within the response to get the JWKS doc. One other characteristic, Service Account Token Quantity Projection, initiatives a service account token right into a pod in a mounted quantity. This service account token will then be despatched to AWS in alternate for an AWS STS token, which can be utilized to entry AWS sources. Since we configure AWS to belief this OIDC supplier and its public keys, the service account tokens issued by these public keys are additionally trusted by AWS. Therefore AWS can difficulty STS tokens to this OIDC supplier, which is a Kubernetes cluster. Token alternate is completed mechanically by the EKS net identification webhook.

Kubernetes API server and S3 Configuration for OIDC

This text describes particulars about how one can arrange OIDC in a charmed Kubernetes and combine it with AWS. We suppose the Kubernetes has already been deployed on AWS with Juju. Firstly we have to outline OIDC issuer URL and JWKS URL in API server. Usually Kubernetes API server endpoint just isn’t obtainable to the general public Web. Nonetheless they must be publicly accessible for IAM/STS to retrieve OIDC configuration paperwork. As really helpful by the EKS net establish webhook doc, we take the content material of each endpoints from Kubernetes API server and place them in AWS S3 to make them publicly accessible. Earlier than that, we have to put these S3 public endpoint URLs of OIDC paperwork in API server’s flags service-account-issuer and service-account-jwks-uri to let Kubernetes know so it could possibly embrace the brand new URLs within the content material of OIDC paperwork.

For instance, I’ve created a S3 bucket referred to as charmed-k8s-oidc, and a folder referred to as o7k on this bucket to host the OIDC configuration paperwork. The bucket must have ACL public learn and each configuration paperwork have to have ACL public learn too. So each URLs will appear like the next:

https://charmed-k8s-oidc.s3.amazonaws.com/o7k
https://charmed-k8s-oidc.s3.amazonaws.com/o7k/openid/v1/jwks

Then we are able to use juju command to configure them in API server.

juju config kubernetes-control-plane api-extra-args="service-account-issuer=https://charmed-k8s-oidc.s3.amazonaws.com/o7k service-account-jwks-uri=https://charmed-k8s-oidc.s3.amazonaws.com/o7k/openid/v1/jwks"

Wait a bit for Juju to make the change and restart Kubernetes management airplane providers. Subsequent we have to get the content material of config recordsdata and add them to S3. For safety causes, all requests to the API server want a token. We are able to use any legitimate token for this. For instance, we are able to use the token from the default service account.

Firstly we get its secret identify with:

kubectl get sa default -o json | jq -Mr '.secrets and techniques[].identify | choose(incorporates("token"))'

As an illustration if the returned secret identify is default-token-w6vvd, we are able to get its token with:

kubectl get secret default-token-w6vvd -o json | jq -Mr '.knowledge.token' | base64 -d

Subsequent we are able to get the content material of these OIDC configuration paperwork with the token from above output.

curl -k -H "Authorization: Bearer $TOKEN" -k https://API_SERVER:6443/.well-known/openid-configuration
curl -k -H "Authorization: Bearer $TOKEN" -k https://API_SERVER:6443/openid/v1/jwks

Now we are able to add the content material to charmed-k8s-oidc bucket in S3 as following recordsdata:

/o7k/.well-known/openid-configuration
/o7k/openid/v1/jwks

AWS Configuration

After configuring Kubernetes API server and S3, we are able to create an identification supplier in IAM. Particulars about how one can create an identification supplier is out of the scope of this doc and will be present in AWS paperwork. Briefly talking, when creating the identification supplier, its supplier URL factors to our S3 bucket https://charmed-k8s-oidc.s3.amazonaws.com/o7k, and the viewers is sts.amazonaws.com.

Subsequent we have to create an IAM position to outline what this identification supplier can do. This position has a trusted coverage to permit our new establish supplier to imagine this position. We additionally specify within the trusted coverage to solely enable a specific service account or all service accounts in a specific namespace assume this position.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::xxxxx:oidc-provider/charmed-k8s-oidc.s3.amazonaws.com/o7k"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "charmed-k8s-oidc.s3.amazonaws.com/o7k:sub": "system:serviceaccount:default:*"
                }
            }
        }
    ]
}

On this instance we enable all service accounts within the default namespace to imagine this position. Then we are able to add a permissions coverage to this position. For instance, we may give S3 learn solely entry permission to this position.

EKS Pod Id Webhook setup

Now we’ll return to Kubernetes to arrange EKS pod identification webhook. Its newest model depends on cert supervisor, so we have to deploy cert supervisor first.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/obtain/v1.8.2/cert-manager.yaml

Then checkout the supply code from its github venture and run the command under to deploy.

make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:newest

As a result of latest modifications we are able to press ctrl-c once we see ‘Ready for CSR…’. There’s a pull request to repair that however it hasn’t been merged but on the time of writing this text.

Learn how to use ServiceAccount annotations to entry AWS

As much as right here every thing is about up and we are able to do some checks.

Firstly create a service account within the default namespace to imagine the IAM position we created earlier. We assume the position’s identify is K8SS3ReadOnly.

apiVersion: v1
sort: ServiceAccount
metadata:
  identify: s3sa
  namespace: default
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::xxxxxx:position/K8SS3ReadOnly"
    # non-obligatory: Defaults to "sts.amazonaws.com" if not set
    eks.amazonaws.com/viewers: "sts.amazonaws.com"
    # non-obligatory: When set to "true", provides AWS_STS_REGIONAL_ENDPOINTS env var
    #   to containers
    eks.amazonaws.com/sts-regional-endpoints: "true"
    # non-obligatory: Defaults to 86400 for expirationSeconds if not set
    #   Observe: This worth will be overwritten if specified within the pod 
    #         annotation as proven within the subsequent step.
    eks.amazonaws.com/token-expiration: "86400"

Then we create a pod to make use of this service account to entry AWS providers.

apiVersion: v1
sort: Pod
metadata:
  labels:
    run: my-pod
  identify: my-pod
spec:
  serviceAccountName: s3sa
  initContainers:
  - picture: amazon/aws-cli
    identify: my-aws-cli
    command: ['aws', 's3', 'ls', 's3://']
  containers:
  - picture: nginx
    identify: my-pod
    ports:
    - containerPort: 80
  dnsPolicy: ClusterFirst
  restartPolicy: At all times

Within the init container of this pod, it makes use of aws command to record all buckets in S3. For the reason that pod makes use of a service account which might assume a job in IAM, we don’t want to offer any secret key and entry key to the pod. When the pod is operating, we are able to use kubectl logs to examine the output of that init container, the place we must always see a listing of buckets in S3.

$ kubectl logs my-pod -c my-aws-cli
2022-09-27 05:09:44 charmed-k8s-oidc
......

On-prem Setup

Lastly, if the Kubernetes cluster just isn’t operating on AWS, e.g. in an on-prem knowledge centre, it’s essential to specify a default area in EKS webhook plugin. To try this, edit Kubernetes deployment pod-identity-webhook and add “- –aws-default-region=[region-name, e.g. us-east-1]” to /webhook command as an argument.

$ kubectl get deploy pod-identity-webhook -o yaml
......
    spec:
      containers:
      - command:
        - /webhook
        - --in-cluster=false
        - --namespace=default
        - --service-name=pod-identity-webhook
        - --annotation-prefix=eks.amazonaws.com
        - --token-audience=sts.amazonaws.com
        - --aws-default-region=us-east-1
        - --logtostderr
......

That is to inform the plugin which AWS area it ought to use. If the Kubernetes cluster is operating on AWS, the webhook plugin will use the area it’s operating on by default. But when the cluster just isn’t operating on AWS, it doesn’t know which area to make use of subsequently we have to specify it within the deployment configuration.