Almost right after Hetzner announced their ARM offering on 12th April 2023 I started to experiment with new infrastructure approach. Note: get 20EUR for your project via this referral - just in case you want to play around.
It’s been over a year since I started to play around with Nomad and Nomad served me well. CSI support become more stable and Nomad itself become more mature in terms of feature set (native service discovery and support for encrypted secrets were nice).
Unfortunately, the ecosystem was kinda lacking - levant was left in a weird state when it was kinda supported, but kinda replaced by nomad-pack - which I guess is kinda a Helm for Nomad?
I love Hashicorp’s work, but I think they are stretched thin in Nomad-space (and do we really need a waypoint?) and after running into more various issues I decided it’s time for another redo! Because this time I will do it right. Right?
I was reluctant for years towards Kubernetes due to its complexity and resource-heaviness. I was fiddling with k3s around my experiments with Nomad, but the overall experience then was not great at that time.
I decided to give it another go by using my new ARM-powered VM from Hetzner. I gave up on High-Avability tho - it’s just not worth the effort for side/home projects. Some things eventually will go sideways and restoring some weird state on 1⁄3 third of the servers during the weekend is the last thing I want to be doing nowadays.
Kubernetes-quirks aside some takeaways regarding moving to ARM:
majority of the ecosystem seems to be just there. I guess Apple with their M1 really pushed the ecosystem for daily open-source consumers like myself
you will have a hard time with truly outdated dependencies - who would guess that unmaintained software will just rot? If only one could predict that; I had to replace some ancient ElasticSearch version with zincsearch and plug it into an existing Rails app using some forked gem, ugh
keeping good practices - IaC, CI/CD pipelines, auto-dependencies upgrades - made the transition so smoother. Once again - who would guess that regular maintenance would make things easier? I feel like I really should write a book about it or something. Or at least record a podcast episode.
building docker images on Github actions via QEMU is shaite for anything bigger than a “hello world” project (or you can say - it’s extremly slow); one of my build was running for 40+ min (!); do yourself a favor and check out CircleCI ARM offering - simply moving workflow there gave me 20x performance boost
Aside few hiccups my overall experience is truly positive so expect some k3s/ARM infra-related blog posts coming up!