New infrastructure - recap

So I started to play around with Hasicorp Nomad, and for the time being, I’m done (well, not really).

I have built the new infrastructure around Nomad and tooling provided by Hetzner Cloud - cloud servers (as I was already using those anyway), their load balancer, internal networking, firewall functionality, and attachable data volumes.

Advertisement: If you would like to play around with Hetzner - get 20EUR for new Hetzner Cloud account via this referral link

It took me like a month to move everything - I’m still fixing some issues - but overall, it was pretty fun!

The big picture

1. I’m using Cloudflare / OVH for managing DNS. Cloudflare also acts as a reverse proxy/CDN by its nature.
2. I’m using a single Hetzner Load Balancer that is exposed to an external network and connected to an internal Hetzner Network along with all my cloud servers. Everything except a single SSH port is behind iptables and behind firewall rules provided by Hetzner (because I don’t trust my iptables skills :P). Internal network is considered as trusted and all traffic is allowed there.
3. On each host, I have traefik running. Hetzner Load Balancer is handling certificates tho (using ACME DNS challenge) - traefik uses consul and services tags to handle routing (port :443 on balancer -> goes to :80 on the nodes and that goes to the traefik).

Single node details

1. No magic here, the latest Ubuntu LTS bootstrapped with some Ansible playbooks.
2. Every server has consul, vault and Nomad installed.
3. Nomad acts as both server and a client (runs jobs) - which works perfectly fine for my needs.
4. Because the CSI drivers are absolutely not reliable, some services are bound to particular nodes - it’s a less than ideal workaround for now.

How-to and tools

I quickly realized that the how part is too big to cover in a single blog post, and I hope to follow up on the detailed setup, quirks, and gotchas in the upcoming weeks. First, I need to think about a good starting point as there is a lot to cover here!

Everything was set up using Ansible (cloud servers part) and Terraform (infrastructure - servers & services). Every project runs as a docker container, and almost everything has a CI configured via Github Actions.