Provisioning OCI Load Balancers for Kubernetes Services of Type LoadBalancer

oci.oraclecloud.com/load-balancer-type: "lb"

Note that lb is the default value of the oci.oraclecloud.com/load-balancer-type annotation. If you do not explicitly include the annotation in the service definition, the default value of the annotation is used.

For example, consider the following configuration file, nginx_lb.yaml. It defines a deployment (kind: Deployment) for the nginx app, followed by a definition of a service of type of LoadBalancer (type: LoadBalancer) that balances http traffic on port 80 for the nginx app.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

The first part of the configuration file defines an Nginx deployment, requesting that it be hosted on 3 pods running the nginx:latest image, and accept traffic to the containers on port 80.

The second part of the configuration file defines the Nginx service, which uses type LoadBalancer to balance Nginx traffic on port 80 amongst the available pods.

To create the deployment and service defined in nginx_lb.yaml while connected to your Kubernetes cluster, enter the command:

kubectl apply -f nginx_lb.yaml

This command outputs the following upon successful creation of the deployment and the load balancer:

deployment "my-nginx" created
service "my-nginx-svc" created

The load balancer may take a few minutes to go from a pending state to being fully operational. You can view the current state of your cluster by entering:

kubectl get all

The output from the above command shows the current state:

NAME                                  READY     STATUS    RESTARTS   AGE
po/my-nginx-431080787-0m4m8           1/1       Running   0          3m
po/my-nginx-431080787-hqqcr           1/1       Running   0          3m
po/my-nginx-431080787-n8125           1/1       Running   0          3m

NAME               CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
svc/kubernetes     203.0.113.1     <NONE>           443/TCP        3d
svc/my-nginx-svc   203.0.113.7     192.0.2.22       80:30269/TCP   3m

NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/my-nginx           3         3         3            3           3m

NAME                            DESIRED   CURRENT   READY     AGE
rs/my-nginx-431080787           3         3         3         3m

The output shows that the my-nginx deployment is running on 3 pods (the po/my-nginx entries), that the load balancer is running (svc/my-nginx-svc) and has an external IP (192.0.2.22) that clients can use to connect to the app that's deployed on the pods.

When Container Engine for Kubernetes provisions a load balancer for a Kubernetes service of type LoadBalancer, you can specify that you want to terminate SSL at the load balancer. This configuration is known as frontend SSL. To implement frontend SSL, you define a listener at a port such as 443, and associate an SSL certificate with the listener.

Note that you can implement full point-to-point SSL encryption between clients and application pods running on worker nodes. To do so, create a load balancer with SSL termination (as described in this section), and also associate an SSL certificate with the load balancer's backend set (see Implementing SSL/TLS between the Load Balancer and Worker Nodes).

This example provides a walkthrough of the configuration and creation of a load balancer with SSL support.

Consider the following configuration file, nginx-demo-svc-ssl.yaml, which defines an Nginx deployment and exposes it via a load balancer that serves http on port 80, and https on port 443. This sample creates an Oracle Cloud Infrastructure load balancer, by defining a service with a type of LoadBalancer (type: LoadBalancer).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: nginx-service
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/oci-load-balancer-tls-secret: ssl-certificate-secret
spec:
  selector:
    app: nginx
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80

The Load Balancer's annotations are of particular importance. The ports on which to support https traffic are defined by the value of service.beta.kubernetes.io/oci-load-balancer-ssl-ports. You can declare multiple SSL ports by using a comma-separated list for the annotation's value. For example, you could set the annotation's value to "443, 3000" to support SSL on ports 443 and 3000.

The required TLS secret, ssl-certificate-secret, needs to be created in Kubernetes. This example creates and uses a self-signed certificate. However, in a production environment, the most common scenario is to use a public certificate that's been signed by a certificate authority.

The following command creates a self-signed certificate, tls.crt, with its corresponding key, tls.key:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"

Now that you created the certificate, you need to store both it and its key as a secret in Kubernetes. The name of the secret must match the name from the service.beta.kubernetes.io/oci-load-balancer-tls-secret annotation of the load balancer's definition. Use the following command to create a TLS secret in Kubernetes, whose key and certificate values are set by --key and --cert, respectively.

kubectl create secret tls ssl-certificate-secret --key tls.key --cert tls.crt

You must create the Kubernetes secret before you can create the service, since the service references the secret in its definition. Create the service using the following command:

kubectl create -f manifests/demo/nginx-demo-svc-ssl.yaml

Watch the service and wait for a public IP address (EXTERNAL-IP) to be assigned to the Nginx service (nginx-service) by entering:

kubectl get svc --watch

The output from the above command shows the load balancer IP to use to connect to the service.

NAME            CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
nginx-service   192.0.2.1      198.51.100.1     80:30274/TCP   5m

The load balancer is now running, which means the service can now be accessed as follows:

• using http, by entering:

  curl http://198.51.100.1

• using https, by entering:

  curl --insecure https://198.51.100.1

  The "--insecure" flag is used to access the service using https due to the use of self-signed certificates in this example. Do not use this flag in a production environment where the public certificate was signed by a certificate authority.

Note: When a cluster is deleted, a load balancer that's dynamically created when a service is created will not be removed. Before deleting a cluster, delete the service, which in turn will result in the removal of the load balancer. The syntax for this command is:

kubectl delete svc SERVICE_NAME

For example, to delete the service from the previous example, enter:

kubectl delete svc nginx-service

When Container Engine for Kubernetes provisions a load balancer for a Kubernetes service of type LoadBalancer, you can specify that you want to implement SSL between the load balancer and the backend servers (worker nodes) in the backend set. This configuration is known as backend SSL. To implement backend SSL, you associate an SSL certificate with the load balancer's backend set.

Note that you can implement full point-to-point SSL encryption between clients and application pods running on worker nodes. To do so, associate an SSL certificate with the load balancer's backend set (as described in this section), and also create a load balancer with SSL termination (see Terminating SSL/TLS at the Load Balancer).

To specify the certificate to associate with the backend set, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-tls-backendset-secret: <value>

where <value> is the name of a Kubernetes secret you've created to contain a signed certificate and the private key to the certificate. Note that you must create the Kubernetes secret before you can create the service, since the service references the secret in its definition.

The following example creates and uses a self-signed certificate, which is usually acceptable for internal communication between the load balancer and the backend set. However, if you prefer, you could use a public certificate that's been signed by a certificate authority.

For example:

1. Generate a private key by entering:

   openssl genrsa -out ca.key 2048

2. Generate a certificate by entering:

   openssl req -x509 -new -nodes -key ca.key -subj "/CN=nginxsvc/O=nginxsvc" -days 10000 -out ca.crt

3. Store the certificate and the key as a secret in Kubernetes by entering:

   kubectl create secret generic ca-ser-secret --from-file=tls.crt=tls.crt --from-file=tls.key=tls.key --from-file=ca.crt=ca.crt
4. Define an Nginx deployment and expose it via a load balancer that serves http on port 80, and https on port 443. This sample creates an Oracle Cloud Infrastructure load balancer, by defining a service with a type of LoadBalancer (type: LoadBalancer).

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-deployment
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx
           ports:
           - containerPort: 80
   ---
   apiVersion: v1
   metadata:
     name: nginx-service
     annotations:
       oci.oraclecloud.com/load-balancer-type: "lb"
       service.beta.kubernetes.io/oci-load-balancer-tls-backendset-secret: ca-ser-secret
       service.beta.kubernetes.io/oci-load-balancer-ssl-ports: "443" 
   spec:
     selector:
       app: nginx
     type: LoadBalancer
     ports:
     - name: http
       port: 80
       targetPort: 80
     - name: https
       port: 443
       targetPort: 443

Communication between the load balancer and the worker nodes in the backend set is encrypted using the key and certificate stored in the ca-ser-secret Kubernetes secret that you created earlier.

The shape of an Oracle Cloud Infrastructure load balancer specifies its maximum total bandwidth (that is, ingress plus egress). By default, load balancers are created with a shape of 100Mbps. Other shapes are available, including 400Mbps and 8000Mbps.

To specify an alternative shape for a load balancer, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-shape: <value>

where value is the bandwidth of the shape (for example, 100Mbps, 400Mbps, 8000Mbps).

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-shape: 400Mbps
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Sufficient load balancer quota must be available in the region for the shape you specify. Enter the following kubectl command to confirm that load balancer creation did not fail due to lack of quota:

kubectl describe service <service-name>

Note that Oracle recommends you implement Kubernetes services of type LoadBalancer as cost-efficient flexible load balancers rather than as fixed-shape (dynamic) load balancers (see Specifying Flexible Load Balancer Shapes).

The shape of an Oracle Cloud Infrastructure load balancer specifies its maximum total bandwidth (that is, ingress plus egress). As described in Specifying Alternative Load Balancer Shapes, you can specify different load balancer shapes.

In addition, you can also specify a flexible shape for an Oracle Cloud Infrastructure load balancer, by defining a minimum and a maximum bandwidth for the load balancer.

To specify a flexible shape for a load balancer, add the following annotations in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "<min-value>"
service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "<max-value>"

where:

• "<min-value>" is the minimum bandwidth for the load balancer, in Mbps (for example, "10")
• "<max-value>" is the maximum bandwidth for the load balancer, in Mbps (for example, "100")

Note that you do not include a unit of measurement when specifying bandwidth values for flexible load balancer shapes (unlike for pre-defined shapes). For example, specify the minimum bandwidth as 10 rather than as 10Mbps.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-shape: "flexible"
    service.beta.kubernetes.io/oci-load-balancer-shape-flex-min: "10"
    service.beta.kubernetes.io/oci-load-balancer-shape-flex-max: "100"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

When provisioning an Oracle Cloud Infrastructure load balancer for a Kubernetes service of type LoadBalancer, you can specify the maximum idle time (in seconds) allowed between two successive receive or two successive send operations between the client and backend servers.

To explicitly specify a maximum idle time, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-connection-idle-timeout: <value>

where value is the number of seconds.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-connection-idle-timeout: 100
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

Note that if you don't explicitly specify a maximum idle time, a default value is used. The default value depends on the type of listener:

• for TCP listeners, the default maximum idle time is 300 seconds
• for HTTP listeners, the default maximum idle time is 60 seconds

When Container Engine for Kubernetes provisions a load balancer for a Kubernetes service of type LoadBalancer, you can define the type of traffic accepted by the listener by specifying the protocol on which the listener accepts connection requests.

Note that if you don't explicitly specify a protocol, "TCP" is used as the default value.

To explicitly specify the load balancer listener protocol when Container Engine for Kubernetes provisions a load balancer for a Kubernetes service of type LoadBalancer, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-backend-protocol: <value>

where <value> is the protocol that defines the type of traffic accepted by the listener. For example, "HTTP". Valid protocols include "HTTP" and "TCP".

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-backend-protocol: "HTTP"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx

You can use the security list management feature to configure how security list rules are managed for an Oracle Cloud Infrastructure load balancer that Container Engine for Kubernetes provisions for a Kubernetes service of type LoadBalancer. This feature is useful if you are new to Kubernetes, or for basic deployments.

To specify how the Kubernetes security list management feature manages security lists when Container Engine for Kubernetes provisions a load balancer for a Kubernetes service of type LoadBalancer, add the following annotation in the metadata section of the manifest file:

service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: <value>

where <value> is one of:

• "None": (recommended) No security list management is enabled. You have to set up a security rule that allows inbound traffic to the appropriate ports for node port ranges, the kube-proxy health port, and the health check port ranges. Additionally, you have to set up security rules to allow inbound traffic to load balancers (see Security Rules for Load Balancers and Network Load Balancers).
• "All": (default) All required security list rules for load balancer services are managed.
• "Frontend": Only security list rules for ingress to load balancer services are managed. You have to set up a security rule that allows inbound traffic to the appropriate ports for node port ranges, the kube-proxy health port, and the health check port ranges.

Oracle recommends that you explicitly set service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode to None.

In clusters with managed nodes, if you don't explicitly specify a management mode or you specify an invalid value, all security list rules are managed (equivalent to "All"). Be aware that in this case, Container Engine for Kubernetes creates a security rule that allows inbound traffic from 0.0.0.0/0 (or from the source port ranges specified in the manifest file) to listener ports. In clusters with virtual nodes, security list management is never enabled and you always have to manually configure security rules (equivalent to "None").

Note that there are limits to the number of ingress and egress rules that are allowed in a security list (see Security List Limits). If the number of ingress or egress rules exceeds the limit, and <value> is set to "All" or "Frontend", creating or updating the load balancer fails.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-nginx-svc
  labels:
    app: nginx
  annotations:
    oci.oraclecloud.com/load-balancer-type: "lb"
    service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: "Frontend"
spec:
  type: LoadBalancer
  ports:
  - port: 80
  selector:
    app: nginx