Hello World!

21st August 2018 1 By Jonny

In my last post I had created a gogs instance using a MariaDB backend running on my k8s cluster. I had finished up by making the gogs HTTP service available to my internal network using a NodePort service in k8s.

To the router … and beyond

When a service is exposed using a NodePort in k8s it is assigned a ‘high’ TCP port (30000-32767) which is mapped to the origin port of the pod. Using high ports, whilst convenient is not ideal, as given this is HTTP traffic, port 80 or 443 would make more sense. The service is also exposed on every node in the k8s cluster, which again is convenient as it removes a single point of failure, and means the service isn’t tied to a single IP address.

Up in the cloud having a high port exposed on the cluster nodes is not a problem, as typically a load balancer or similar solution would be used. As I am not using a cloud, and my home lab is fairly cheap, I’ll need a similar (but different) solution. Fortunately there are quite a few options available to me. I could run a reverse proxy web server such as Apache or NGINX which would provide the functionality I require. In the end I settled on using HAProxy on my NAS system (which runs CentOS 7). There was no real reason to choose HAProxy over Apache or NGINX other than the requirement I have is exactly what HAProxy is designed for and it wasn’t a utility I’d used before, so it offered a learning opportunity.

HAProxy Configuration

There are many ways that HAProxy can be set up, however I settled on using a backend and frontend configuration. In the backend section I am able to define the hosts that will be providing the services and choose which load balancing algorithm to use. The frontend section allows me to choose which backend (if I have multiple backends defined) to use depending upon certain conditions being met, and also the port on which the HAProxy will listen.

Backend Configuration

In my environment I have the HAProxy listen on both port 80 (HTTP) and 443 (HTTPS). Starting with the backend, the configuration looks as follows:

backend gogs
mode http
balance source
option forwardfor
cookie SRV_ID prefix
server green green.kube.ipa.champion:31557 cookie 2 check
server orange orange.kube.ipa.champion:31557 cookie 3 check
server pink pink.kube.ipa.champion:31557 cookie 4 check
server white white.kube.ipa.champion:31557 cookie 5 check

The backend service will use the HTTP mode (which really means L7 load balancing which provides more control/options for load balancing compared to TCP mode (which is L4 load balancing). I’ve chosen to use ‘source‘ as my load balancing algorithm, which should mean that each client should always use the same backend server. The option ‘forwardfor‘ should mean that HAProxy passes the original source IP to the backend as the requesting IP address rather than the IP address of the HAProxy, which should make the logs more understandable should any problems occur. Finally, I’m setting a cookie prefix based on the server ID, which can be useful for debugging should I have any errors.

I have also listed out four end nodes to provide the gogs service – I have dropped off my ‘blue’ node as it is the k8s master, and by not being an available backend the load on this server can be reduced. I have also specified that each backend node will set a cookie and will be checked by HAProxy to ensure it remains live.

Frontend Configuration

A backend is no good without a frontend. Starting with the port 80 (HTTP) service, I have defined a front end as follows:

frontend web
bind *:80
mode http
reqadd X-Forwarded-Proto:\ http

acl dst_gogs hdr(host) -i gogs.ipa.champion
use_backend gogs if dst_gogs

This configuration means that HAProxy will listen on port 80 (so I need to make sure this particular server has nothing else listening on port 80 already). The frontend is configured to use http mode also and we are (again – possibly not required) specifying that the source IP is added to the header.

I have created an access control list (ACL) that is telling HAProxy to use my defined gogs backend if the requested destination is gogs.ipa.champion (which is my internal fully qualified domain name for the gogs service). I could define a default_backend to use if no matching rule is found, but for now I’ll leave it as it is.

The HAProxy service is then started and enabled on my CentOS 7 NAS system, and the firewall opened for port 80 traffic.

systemctl enable haproxy && systemctl start haproxy
firewall-cmd --add-service http && firewall-cmd --add-service http --perm

If all has gone well (and I have set up my home network DNS to point gogs.ipa.champion to my CentOS 7 NAS system, then when I browse to http://gogs.ipa.champion I’ll see the gogs front page:

You’ll need to look closely, but that is http://gogs.ipa.champion

For those with good eyesight, you can probably make out that this is a Chrome incognito window, browsing to the non-secure site http://gogs.ipa.champion … which would be great if this were 2005 and we didn’t care about TLS and external access.

Securing and expanding our reach

Like most home Internet connections, my ISP blocks port 80 – after all, we don’t want any old Tom, Dick, or Harry running a web server from home. However, we can have a Jonny run a TLS secured web server from home, as they haven’t (yet) blocked port 443. Given that Chrome, Firefox (and I expect Edge and Safari too) will start flagging up warnings about non-secure sites, and presumably eventually default to HTTPS connections, then if I want to make my site accessible externally I should secure it via TLS and expose it via port 443 on my router.

It’s easy enough to port forward my public router IP port 443 to my CentOS NAS system port 443. I’ll now need to configure my HAProxy to also listen on port 443, and most importantly configure it to provide TLS certificates to secure the remote client to HAProxy traffic. I will be leaving the HAProxy to k8s cluster traffic as unencrypted as this traffic is all internal to my network and should be seen as ‘trusted’ and therefore not requiring encryption. It is possible to encrypt this traffic if that is so desired though.

I have an additional frontend section to my HAProxy configuration file as follows:

frontend web-ssl
bind *:443 ssl crt /etc/haproxy/blog.ervine.org.pem ssl crt /etc/haproxy/gogs.ipa.champion.pem reqadd X-Forwarded-Proto:\ https
mode http

use_backend gogs if { hdr(host) -i gogs.ervine.org }

This is a similar configuration to the non-secure frontend listed above. The main changes here are that we are listening on port 443, we have specified a TLS certificate to use to secure the traffic, and I’ve made a change to the ACL to reflect the external domain name that my servers will use (ervine.org).

I had originally configured this to use a ‘self-signed’ TLS certificate, which will secure the traffic between client and HAProxy, but will also result in warnings being flagged up by the browser that the web server certificate is not trusted.

TLS certificates

In the past trusted TLS/SSL certificates were expensive to purchase and needed to be renewed on an annual basis. Nowadays TLS encryption is freely available (with a bit of technical know-how) through a service called letsencrypt. When this service first started it could be a bit complicated to get your own certificate minted, however there are now tools that make this easy and straightforward with the following caveats:

  • You have administrative access to your own domain

Yep – that’s pretty much it. You just need your own domain (some routers will integrate with letsencrypt directly, although I suspect this would be limited to a specific hostname). Managing my own domain means I can create DNS records as necessary – which is important for the method of domain ownership verification for the certificate generation process.

The CentOS EPEL repository contains the certbot package which is a set of tools that help to simplify and automate the letsencrypt certificate process. I install this package on my CentOS NAS system first of all.

$ sudo yum install -y certbot

Once this is installed, I’ll then use the certbot-2 utility to request a certificate for gogs.ervine.org as follows:

$ certbot-2 certonly --manual --preferred-challenges dns
...
Please enter in your domain name(s) (comma and/or space separated) (Enter 'c' to cancel): gogs.ervine.org
...
IMPORTANT NOTES:
Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/gogs.ervine.org/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/gogs.ervine.org/privkey.pem
Your cert will expire on 2018-11-19. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew all of your certificates, run
"certbot renew"
...

The certbot process asks that a TXT record be created in DNS with a particular value. The certbot process will then read this record and ensure that it matches the value provided. If so, then the domain ownership has been confirmed and the certificate is generated. The only drawback to letsencrypt certificates is that they need to be renewed every 90 days rather than every year. However, the command to renew the certificates is very simple, and can easily be run as a cron job – which should mean the certificate is renewed automatically.

Creating the certificate pem file for HAProxy

The final step in this process is to create the certificate pem file that HAProxy expects which contains both the certificate chain and the private key in a single file. This is very straightforward as well:

$ sudo cp /etc/letsencrypt/live/gogs.ervine.org/fullchain.pem /etc/haproxy/gogs.ervine.org.pem
$ sudo cat /etc/letsencrypt/live/gogs.ervine.org/privkey.pem >> /etc/haproxy/gogs.ervine.org.pem

$ sudo systemctl restart haproxy

The combined file is simply the private key ‘catted’ into the certificate file. Restarting HAProxy should mean that it is now using the TLS certificate file, and all being well, the gogs service should now be available at https://gogs.ervine.org from external sources, with a valid TLS certificate being presented.

A letsencrypt secured home web server

And there we have it! A letsencrypt TLS secured home web server – offering a gogs service hosted on an internal k8s cluster.

Summary

In this post I’ve made my k8s hosted gogs service available to my home network over port 80 using an HAProxy. I’ve then expanded the configuration of HAProxy to also offer a TLS secured version of gogs. As a final touch, I’ve used a letsencrypt TLS certificate to provide a trusted TLS certificate to the gogs service and made this available to remote clients over the Internet.

Next time – I’ll step back a bit and look to create some more container images locally that I can then use on my k8s cluster with a view to removing any dependency on remote container repositories.