So I started to play around with Hasicorp Nomad, and for the time being, I’m done (well, not really).
I have built the new infrastructure around Nomad and tooling provided by Hetzner Cloud - cloud servers (as I was already using those anyway), their load balancer, internal networking, firewall functionality, and attachable data volumes.
Advertisement: If you would like to play around with Hetzner - get 20EUR for new Hetzner Cloud account via this referral link
It took me like a month to move everything - I’m still fixing some issues - but overall, it was pretty fun!
The big picture
- I’m using Cloudflare / OVH for managing DNS. Cloudflare also acts as a reverse proxy/CDN by its nature.
- I’m using a single Hetzner Load Balancer that is exposed to an external network and connected to an internal Hetzner Network along with all my cloud servers. Everything except a single SSH port is behind iptables and behind firewall rules provided by Hetzner (because I don’t trust my iptables skills :P). Internal network is considered as trusted and all traffic is allowed there.
- On each host, I have traefik running. Hetzner Load Balancer is handling certificates tho (using ACME DNS challenge) - traefik uses consul and services tags to handle routing (port
:443on balancer -> goes to
:80on the nodes and that goes to the traefik).
Single node details
- No magic here, the latest Ubuntu LTS bootstrapped with some Ansible playbooks.
- Every server has consul, vault and Nomad installed.
- Nomad acts as both server and a client (runs jobs) - which works perfectly fine for my needs.
- Because the CSI drivers are absolutely not reliable, some services are bound to particular nodes - it’s a less than ideal workaround for now.
How-to and tools
I quickly realized that the how part is too big to cover in a single blog post, and I hope to follow up on the detailed setup, quirks, and gotchas in the upcoming weeks. First, I need to think about a good starting point as there is a lot to cover here!
Everything was set up using Ansible (cloud servers part) and Terraform (infrastructure - servers & services). Every project runs as a docker container, and almost everything has a CI configured via Github Actions.