Creating a Custom NGINX Ingress controller Kubernetes

This is a step by step tutorial on creating a secure NGINX server with custom identity, error pages, CORS policies, Enhanced cipher suites, SSL TLS compatibilities.

Introduction

I was avoiding writing several long YAML files to make this happen, but once you get used to playing around with K8s, you discover the power of orchestrating an application in a scalable way. You do have to write a docker-compose file in the world of Docker but I have found writing YAML for Kubernetes more tedious but it might be worth it once you understand what more you can do with K8S.

It’s probably worth a quick introduction to clear things up and like everyone the first thing I learned in Kubernetes is to create a deployment that serves a website and created a service around it. The first question that came in my mind afterward is how these websites are served? don’t they need a web-server? I figured it was the nginx behind it. but unlike normal Nginx in a docker environment, it was the Nginx ingress controller that did the job of routing the requests. Traditionally, you would create a LoadBalancer service for each public system you want to expose. This can get rather expensive.(for the IPs). Ingress gives you a way to route requests to services based on the request host or path, centralizing a number of services into a single entrypoint.

Note. Many people get confused between using the community based ingress controller and the official nginx ingress controller

We are using the official nginxinc repository instead of Kubernetes community based repository. To know the exact differences these make, follow the link here.

Now that we know what an ingress controller is, Lets tweak the backend to suit our needs. at first, I was tasked to hide the server name and the server tokens. Should be Easy right? ….. NO. You CAN’T hide the server name without a nginx plus subscription.

So what we did is find a module that helps to hide the nginx server name and provide custom headers. Now, How do we add the third party nginx modules?

The above Dockerfile will compile the nginx source binaries with the third party module and creates a container image.

To enable the third party module, add this line

load_module modules/ngx_http_headers_more_filter_module.so;

to the nginx.conf and to hide the server name and tokens, add these parameters in the server block of nginx.default.conf

    server_tokens off;
more_set_headers 'Server: YOURSERVERNAME';

here is the catch, this only does help in providing the headers that are hidden in the Network tab. if you happen to encounter an error, the default error pages through nginx will be shown and your tedious work is of no use.

SO create and add beautiful custom error pages of your choice and serve it with the location blocks.

Once that is done, Push this image to your public or private repository and use it on your nginx-ingress deployment.

Setting Up CORS

As spoken earlier, tightening the security for a web app is a must. To do that, we need a strong set of CORS policies in place. To understand in-depth about the CORS, Read it here.

This is a courtesy of countless hours of searching through internet and the solution is from one of the developers of nginx ingress controller. Here.

For example, depending on an application, the user might want to customize:

  • Access-Control-Allow-Methods
  • Access-Control-Allow-Headers
  • Access-Control-Expose-Headers
  • Access-Control-Allow-Credentials
  • Access-Control-Max-Age
  • the behavior when a request comes from an origin that is not allowed. For example, the user might want to return a 401 with a custom error page.
  • other possible customizations.

To be able to customize those items, the user will have to change the ingress.tmpl, which undermines the value of having CORS annotations.

We can try to provide the necessary level of customization by adding additional annotations. However, I don’t think it would be the best solution, because CORS requirements can be quite different and it is possible it would require adding more and more annotations to cover those requirements.

As an alternative, I can suggest the following solution that utilizes the ConfigMaps resource and the nginx.org/location-snippets annotations:

Create a configMap with the CORS configuration

Create a file named cors.conf with the following content (the content is based on your configuration from ingress.tmpl, but for simplicity, it doesn't have the origin check):

set $cors "true";
set $corsmethod "${cors}nonoptions";
if ($request_method = 'OPTIONS') {
set $corsmethod "${cors}options";
}
if ($corsmethod = "truenonoptions") {
add_header 'Access-Control-Allow-Origin' "$http_origin" always;
add_header 'Access-Control-Allow-Credentials' 'true' always;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PATCH, PUT, DELETE, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since';
add_header 'Access-Control-Expose-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Range,Range';
}
if ($corsmethod = "trueoptions") {
add_header 'Access-Control-Allow-Origin' "$http_origin";
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Access-Control-Allow-Methods' 'GET, POST, PATCH, PUT, DELETE, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'Authorization,Content-Type,Accept,Origin,User-Agent,DNT,Cache-Control,X-Mx-ReqToken,Keep-Alive,X-Requested-With,If-Modified-Since';
add_header 'Content-Length' 0;
add_header 'Content-Type' 'text/plain charset=UTF-8';
return 204;
}

Create a ConfigMap resource from cors.conf:

$ kubectl create configmap cors-config --from-file=./cors.conf

Step 3.2. Deploy the Ingress controller and mount the ConfigMap resource on the file system, using the following manifest:

apiVersion: v1
kind: deployment
metadata:
name: nginx-ingress
labels:
app: nginx-ingress
spec:
replicas: 1
selector:
app: nginx-ingress
template:
metadata:
labels:
app: nginx-ingress
spec:
containers:
- image: "Your Custom Nginx image here "
imagePullPolicy: Always
name: nginx-ingress
ports:
- containerPort: 80
hostPort: 80
- containerPort: 443
hostPort: 443
args:
- -v=3
volumeMounts:
- name: config-volume
mountPath: /etc/nginx/custom-snippets/
volumes:
- name: config-volume
configMap:
name: cors-config

Note that cors.conf file will be placed in the /etc/nginx/custom-snippets folder.

Using the nginx.org/location-snippets annotation, include cors.conf in location blocks of your Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.org/location-snippets: "include /etc/nginx/custom-snippets/cors.conf;"
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80

Create the Ingress resource.
The generated config will look like the following:

...
location /tea {
proxy_http_version 1.1;
include /etc/nginx/custom-snippets/cors.conf;
...

Using the ConfigMap resource and the nginx.org/location-snippets annotation allows you to easily add complex NGINX configuration snippets to your Ingress resources. Alternatively, it is possible to put the content of cors.conf into the nginx.org/location-snippetsas shown here.

Why do we need a strong set of cipher suite in nginx? to mitigate against malicious attacks.

Read more about the vulnerabilities here

Edit the configmap for the nginx-ingress and add these following set of values in the data field,

data:
hsts: "True"
hsts-include-subdomains: "True"
hsts-max-age: "31536000"
ssl-ciphers: EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256
EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL
!eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS
ssl-prefer-server-ciphers: "True"
ssl-protocols: TLSv1.2 TLSv1.3

these are some of the suggested set of strong cipher-suites. To check if your SSL, TLS is good and has an “A” rating , visit this site and test your domain.

That all folks!! Thanks for the support!

Photo by Alexander Sinn on Unsplash

I’m gonna keep writing more dev-ops stuff like this.

Thanks to My team, my mentors and colleagues for encouraging us to write and show the world on How we do DevOps ;)

A wanderer lost in the deep sea of stars.