Finishing off

13th September 2018 0 By Jonny

Having detailed how I am continually building and publishing my container images, the final post in this series will round off detailing the WordPress installation that this blog is being hosted on.

Putting it together

It would be very simple to dump all the components needed for WordPress into one large container. However, this wouldn’t really be making the most of using containers, with one of the main advantages being able to break down muti-part application stacks into their component parts. A wordpress installation can be broken down into requiring a database, a PHP processor, and a web server to serve the content. Fortunately, there are container images for all of these. I am also able to re-use my pre-existing MariaDB container which is currently serving the gogs instance.

I’m not too sure whether this is good practice or not to use a single container application (MariaDB in this case) for multiple other applications. In a non-containerised environment this would be perfectly normal though.

I have chosen to use the Alpine Linux based containers for PHP 7.2 with the latest wordpress installation present as well as the latest NGINX container image. As before, I have also chosen to build these images in my home lab rather than pull the images from the hub.docker.com website. As expected these builds are all added to Jenkins and are built on a daily basis and published to my docker registry.

PHP 7.2 + wordpress in more detail

The original container image is from the Docker hub, and I have made no changes to the repository, other than basing the original images from my own internal builds. The deployment to the kubernetes cluster involves a deployment of the image and also a persistent volume to hold the PHP configuration and a persistent volume to hold the wordpress files. Up until now the persistent volumes being used have had a policy of ReadWriteOnce, which means that only one pod can write to the volume, although many can read from it. For the wordpress files though, I will want them to be written to by the wordpress pod and also the web server pod (at least I think I want/need both pods to be able to write to the volume). This is achieved by setting the PV policy to be ReadWriteMany (RWX), which will allow the same volume to be written by multiple pods. The PV for the PHP configuration can remain as a ReadWriteOnce policy.

The PHP7.2 and wordpress deployment YAML looks as follows:

kind: Deployment
metadata:
name: wordpress
namespace: production
spec:
selector:
matchLabels:
app: wordpress
strategy:
type: Recreate
template:
metadata:
labels:
app: wordpress
spec:
containers:
- image: registry.ervine.org:5000/arm/alpine/wordpress-source
imagePullPolicy: Always
name: wordpress
ports:
- containerPort: 9000
protocol: TCP
name: wordpress-port
env:
# Use secret in real usage
- name: WORDPRESS_DB_HOST
value: mariadb
- name: WORDPRESS_DB_PASSWORD
valueFrom:
secretKeyRef:
name: wp-db-pass
key: wp-db-pass
- name: WORDPRESS_DB_NAME
valueFrom:
secretKeyRef:
name: wp-db
key: wp-db
- name: WORDPRESS_DB_USER
valueFrom:
secretKeyRef:
name: wp-db-user
key: wp-db-user
volumeMounts:
- name: wordpress-pv
mountPath: /var/www/html
- name: wp-php-conf-pv
mountPath: /etc/php7
volumes:
- name: wordpress-pv
persistentVolumeClaim:
claimName: wordpress-pvc
- name: wp-php-conf-pv
persistentVolumeClaim:
claimName: wp-php-conf-pvc

The wordpress instance also relies on a database backend, and I have reused my existing MariaDB pod that is currently being used for gogs. I was, and still remain, uncertain whether this is a good idea or not. The alternative would be to deploy a separate MariaDB (or other DB) pod exclusively for wordpress usage. At the moment, a failure of the MariaDB pod will result in the wordpress and gogs instances being unavailable, and the whole point of a kubernetes architecture is to try and reduce this.

Making it available

On it’s own the wordpress/PHP pod doesn’t do very much, and I need a web browser pod to actually make the wordpress instance available. Below is the deployment yaml for an NGINX web server instance. I chose NGINX as it appears to be reasonably lightweight and, at the moment, is a very popular choice for wordpress. NGINX is also frequently used for kubernetes demos and it made sense to re-use an image I’d previously put together.

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx-wp
name: nginx-wp
namespace: production
spec:
selector:
matchLabels:
app: nginx-wp
strategy:
type: Recreate
template:
metadata:
labels:
app: nginx-wp
spec:
containers:
- image: registry.ervine.org:5000/arm/alpine/nginx-source
imagePullPolicy: Always
name: nginx-wp
ports:
- containerPort: 80
protocol: TCP
name: nginx-wp
volumeMounts:
- mountPath: /var/www/html
name: wordpress-pv
- mountPath: /etc/nginx
name: nginx-conf-pv
volumes:
- name: wordpress-pv
persistentVolumeClaim:
claimName: wordpress-pvc
- name: nginx-conf-pv
persistentVolumeClaim:
claimName: nginx-conf-pvc

The deployments each deploy a single pod, and they share a persistent volume (wordpress-pv) which is mounted into the same location on each pod (/var/www/html). This particular persistent volume has been set up as ReadWriteMany and therefore it allows both the NGINX pod and the wordpress pod to have write access to the contents. It is into this volume that the wordpress archive is extracted.

The NGINX pod also has a persistent volume created that contains the NGINX configuration – specifically the configuration is set up so that all PHP pages are sent to the remote PHP processor, which runs on the wordpress pod, for interpretation. The configuration for NGINX when passing PHP files looks as follows:

location ~ .php$ {
fastcgi_split_path_info ^(.+?.php)(/.*)$;
fastcgi_pass wordpress:9000;
include fastcgi_params;
fastcgi_param SCRIPT_FILENAME /var/www/html$fastcgi_script_name;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_index index.php;
}

As can be seen, the PHP processor is running on a host named wordpress on port 9000 – the remaining directives are simply instructing what to do with the scripts and where they reside. It would be possible to have these in separate locations on the NGINX pod and the wordpress pod, however it is easier to have them in the same location (and they are ultimately the same files, so it makes logical sense to do this too).

Exposing the services

It’s no real surprise that the wordpress deployment has been exposed within k8s as a ClusterIP service. This makes the pod accessible within the cluster via the name of the pod (to other pods in the same namespace). The pod is also accessible to other pods not in the same namespace via the fully qualified domain name (that is internal to kubernetes): wordpress.<namespace>.cluster.local

The NGINX pod is exposed as a NodePort service which is then fronted by an external HAProxy instance so that the web front end is accessible. As has been documented previously, the HAProxy includes TLS certificates originating from LetsEncrypt to provide valid TLS (HTTPS) connectivity to the NGINX front end (which is what you are using here to read this post).

Lessons learnt along the way …

Obviously the big reason for doing a lot of this work was to increase/improve my knowledge of all things kubernetes, and also to gain a better understanding as to how applications can be run on kubernetes and some of the challenges associated with that. It’s fair to say I made a lot of mistakes along the way, and am probably still making mistakes with my deployments and management of the applications on the cluster. However, I do believe I’ll be able to overcome these or workaround them. As no-one actually reads this blog, it’s not exactly a disaster if it simply disappears either.

The first lesson I learnt is to try and understand the various settings that I am blindly copying and pasting into my yaml files. Early on, it was very instructive to know that when I have a persistent volume policy set to ‘Recycle’ when I delete the deployment, this also means that the persistent volume is wiped. This can be very annoying when it’s your MariaDB deployment being deleted with a perfectly decent gogs pod running. I quickly learnt that Recycle policies are great for testing/development environments, but not for those that you want to use. As it is, the Recycle policy is being deprecated in favour of using dynamic volumes. As such, I will need to learn about dynamic provisioning.

In early testing of deploying NGINX and wordpress as separate pods I had basicalyl copied the deployment yaml from NGINX and made minor modifications to the yaml for wordpress. Unfortunately, I didn’t alter the app labels, so they remained the same. This didn’t cause me any problems until I came to expose my wordpress pod as a ClusterIP service. At this point my wordpress instance would result in seemingly random 503 HTTP errors, suggesting that the web browser (or other component) couldn’t connect to a required resource. All problems are easy when you know the answer – but my lack of knowledge meant the root cause took a while for me to discover. Hence why I would be deleting pods and deployments in an attempt to resolve the issue. I can’t remember where I found an answer to a different question that pointed me in the right direction. However, the troubleshooting tool to use was:

kubectl -n <namespace> get endpoints
NAME ENDPOINTS AGE
gogs 10.244.2.146:3000,10.244.2.146:22 41d
mariadb 10.244.1.124:3306 41d
nginx-wp 10.244.1.125:80 35d
wordpress 10.244.2.148:9000,10.244.1.125:9000 35d

Or at least, it was output like that, that had me realise that somehow, my nginx-wp pod was also being listed as an endpoint for wordpress on port 9000 … which was clearly not going to work. It also explained why it would occasionally work as sometimes the request would go to the wordpress pod, and sometimes not. A quick clean up of the deployment yaml files to give each pod separate labels, and a quick re-deploy soon had wordpress firing as expected.

I’ve also tried to keep as much load of the master node of the kubernetes cluster by directing the HAProxy to direct to the worker nodes. This probably doesn’t make a huge difference, and I could mark the master node as unschedulable. If the master node becomes heavily loaded – which it can do being a Raspberry Pi, then admin of the kubernetes cluster becomes nearly impossible. Maybe having multiple masters would be beneficial? 

Going forward I’d like to look into setting up ingress controllers which might mean I don’t need my HAProxy – although the HAProxy does perform a great job in providing TLS encryption to otherwise non-TLS services on kubernetes.

I have also deployed the metrics-server on the cluster as well as the dashboard. However, at the time of writing, these don’t integrate to provide graphical charts of metrics in the dashboard, as the dashboard is still set up to use the deprecated heapster statistics.

Also deployed is a prometheus and grafana monitoring/graphing set of pods to provide some very pretty dashboards. I feel as though with all the pods being deployed I’m reaching the limit of what my 5 x Pi cluster can achieve – and it’s certain that a lot of the available k8s resources assume you’re running an x86_64 based cluster.

Final thoughts

It’s been a lot of fun learning kubernetes, and I’ve managed to move some of my home infrastructure (and create new services) onto kubernetes, whilst learning more about container creation/building, registry hosting, Jenkins automation, and source code control that I previously didn’t know about. In the same time frame, I’ve also managed to pass the Certified Kubernetes Administrator (CKA) exam and I’ve learnt a lot more about OpenShift as well.