Creating a Custom NGINX Ingress controller Kubernetes

This is a step by step tutorial on creating a secure NGINX server with custom identity, error pages, CORS policies, Enhanced cipher suites, SSL TLS compatibilities.

Introduction

I was avoiding writing several long YAML files to make this happen, but once you get used to playing around with K8s, you discover the power of orchestrating an application in a scalable way. You do have to write a docker-compose file in the world of Docker but I have found writing YAML for Kubernetes more tedious but it might be worth it once you understand what more you can do with K8S.

What’s an Ingress controller?

It’s probably worth a quick introduction to clear things up and like everyone the first thing I learned in Kubernetes is to create a deployment that serves a website and created a service around it. The first question that came in my mind afterward is how these websites are served? don’t they need a web-server? I figured it was the nginx behind it. but unlike normal Nginx in a docker environment, it was the Nginx ingress controller that did the job of routing the requests. Traditionally, you would create a LoadBalancer service for each public system you want to expose. This can get rather expensive.(for the IPs). Ingress gives you a way to route requests to services based on the request host or path, centralizing a number of services into a single entrypoint.

Note. Many people get confused between using the community based ingress controller and the official nginx ingress controller

We are using the official nginxinc repository instead of Kubernetes community based repository. To know the exact differences these make, follow the link here.

nginxinc/kubernetes-ingress

Hiding Server Identity

Now that we know what an ingress controller is, Lets tweak the backend to suit our needs. at first, I was tasked to hide the server name and the server tokens. Should be Easy right? ….. NO. You CAN’T hide the server name without a nginx plus subscription.

So what we did is find a module that helps to hide the nginx server name and provide custom headers. Now, How do we add the third party nginx modules?

STEP-1

• Create a Dockerfile that compiles the source binary with the third party modules.

openresty/headers-more-nginx-module

The above Dockerfile will compile the nginx source binaries with the third party module and creates a container image.

STEP-2

To enable the third party module, add this line

load_module modules/ngx_http_headers_more_filter_module.so;

to the nginx.conf and to hide the server name and tokens, add these parameters in the server block of nginx.default.conf

    server_tokens off;
    more_set_headers 'Server: YOURSERVERNAME';

here is the catch, this only does help in providing the headers that are hidden in the Network tab. if you happen to encounter an error, the default error pages through nginx will be shown and your tedious work is of no use.

SO create and add beautiful custom error pages of your choice and serve it with the location blocks.

Once that is done, Push this image to your public or private repository and use it on your nginx-ingress deployment.

Setting Up CORS

STEP-3

As spoken earlier, tightening the security for a web app is a must. To do that, we need a strong set of CORS policies in place. To understand in-depth about the CORS, Read it here.

Cross-Origin Resource Sharing (CORS)

This is a courtesy of countless hours of searching through internet and the solution is from one of the developers of nginx ingress controller. Here.

For example, depending on an application, the user might want to customize:

• Access-Control-Allow-Methods
• Access-Control-Allow-Headers
• Access-Control-Expose-Headers
• Access-Control-Allow-Credentials
• Access-Control-Max-Age
• the behavior when a request comes from an origin that is not allowed. For example, the user might want to return a 401 with a custom error page.
• other possible customizations.

To be able to customize those items, the user will have to change the ingress.tmpl, which undermines the value of having CORS annotations.

We can try to provide the necessary level of customization by adding additional annotations. However, I don’t think it would be the best solution, because CORS requirements can be quite different and it is possible it would require adding more and more annotations to cover those requirements.

As an alternative, I can suggest the following solution that utilizes the ConfigMaps resource and the nginx.org/location-snippets annotations:

Step 3.1

Create a configMap with the CORS configuration

Step 3.1.1

Create a file named cors.conf with the following content (the content is based on your configuration from ingress.tmpl, but for simplicity, it doesn't have the origin check):

set $cors "true";
set $corsmethod "${cors}nonoptions";
if ($request_method = 'OPTIONS') {
    set $corsmethod "${cors}options";
}
if ($corsmethod = "truenonoptions") {
    add_header 'Access-Control-Allow-Origin' "$http_origin" always;
    add_header 'Access-Control-Allow-Credentials' 'true' always;
    add_header 'Access-Control-Allow-Methods' 'GET, POST, PATCH, PUT, DELETE, OPTIONS' always;
    add_header 'Access-Control-Allow-Headers' 'Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since';
    add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
}
if ($corsmethod = "trueoptions") {
    add_header 'Access-Control-Allow-Origin' "$http_origin";
    add_header 'Access-Control-Allow-Credentials' 'true';
    add_header 'Access-Control-Max-Age' 1728000;
    add_header 'Access-Control-Allow-Methods' 'GET, POST, PATCH, PUT, DELETE, OPTIONS';
    add_header 'Access-Control-Allow-Headers' 'Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since';
    add_header 'Content-Length' 0;
    add_header 'Content-Type' 'text/plain charset=UTF-8';
    return 204;
}

Step 3.1.2.

Create a ConfigMap resource from cors.conf:

$ kubectl create configmap cors-config --from-file=./cors.conf

Step 3.2. Deploy the Ingress controller and mount the ConfigMap resource on the file system, using the following manifest:

apiVersion: v1
kind: deployment
metadata:
  name: nginx-ingress
  labels:
    app: nginx-ingress
spec:
  replicas: 1
  selector:
    app: nginx-ingress
  template:
    metadata:
      labels:
        app: nginx-ingress
    spec:
      containers:
      - image: "Your Custom Nginx image here "
        imagePullPolicy: Always
        name: nginx-ingress
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        args:
         - -v=3
        volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/custom-snippets/
      volumes:
        - name: config-volume
          configMap:
            name: cors-config

Note that cors.conf file will be placed in the /etc/nginx/custom-snippets folder.

Step 3.3

Using the nginx.org/location-snippets annotation, include cors.conf in location blocks of your Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cafe-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/location-snippets: "include /etc/nginx/custom-snippets/cors.conf;"
spec:
  rules:
  - host: cafe.example.com
    http:
      paths:
      - path: /tea
        backend:
          serviceName: tea-svc
          servicePort: 80
      - path: /coffee
        backend:
          serviceName: coffee-svc
          servicePort: 80

Create the Ingress resource.
The generated config will look like the following:

...
       location /tea {
                proxy_http_version 1.1;
                include /etc/nginx/custom-snippets/cors.conf;
...

Using the ConfigMap resource and the nginx.org/location-snippets annotation allows you to easily add complex NGINX configuration snippets to your Ingress resources. Alternatively, it is possible to put the content of cors.conf into the nginx.org/location-snippetsas shown here.

CipherSuites, SSL & TLS

Why do we need a strong set of cipher suite in nginx? to mitigate against malicious attacks.

Read more about the vulnerabilities here

Strong SSL Security on nginx

STEP-4

Edit the configmap for the nginx-ingress and add these following set of values in the data field,

data:
    hsts: "True"
    hsts-include-subdomains: "True"
    hsts-max-age: "31536000"
    ssl-ciphers: EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256
      EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL
      !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS
    ssl-prefer-server-ciphers: "True"
    ssl-protocols: TLSv1.2 TLSv1.3

these are some of the suggested set of strong cipher-suites. To check if your SSL, TLS is good and has an “A” rating , visit this site and test your domain.

SSL Server Test

That all folks!! Thanks for the support!

Photo by Alexander Sinn on Unsplash

I’m gonna keep writing more dev-ops stuff like this.

Thanks to My team, my mentors and colleagues for encouraging us to write and show the world on How we do DevOps ;)