Once again I did a full circle and got back to amd64 architecture, technically replacing everything I setup around 3 years ago. This time I will do it right.
After Hetzner rolled out their new cloud offering, it seems like AMD on Hetzner is kinda dead end at this point (don’t quote me on this). So I ended up in a weird situation where I had some old dangling ARM k3s node, some new PoC Coolify ARM-based server and it was time to put it all back together - obviously by setting up a third new project.
What went right
having argocd + kustomize manifests + some ansible sprinkles for initial bootstrap allowed me to quickly re-create foundation of the infrastructure; once I had that, re-LLM-ing refreshed stack with ArgoCD in mind was fairly pleasant (as I only had to scream at the computer to fix those yamls instead of doing it myself)
I had to refresh one of my app as it was full of security holes, as usual best practices like cough test suite and whole CI/CD pipeline (which needed a nudge) made my life much easier
streamlining my other clusters prior to the move - I didn’t have to think how to spin up basic stuff like redis-es or postgres-es
terraform for, you know, avoiding clickops
What went wrong
general software rot; surprise, surprise - not maintained stuff becomes a tangled mess; expired tokens, outdated keys, broken authorizations, dependencies that no longer exist; the resistance to even start thinking about it is huge
amount of dependencies, forked dependencies (what was I thinking 12 years ago?!), frontend dependencies in particular (I will live with Bootstrap 2 till the day I die); some hard to update/migrate software - e.g. had to give up on plausible, didn’t want to deal with 2 -> 3 upgrade, clickhouse - I just scraped everything and moved to umami
terraform; as I had like 3 different repos which managed different parts of infra, and everything eventually got out of date (went through local state, atlantis to terraform cloud route) - but that’s on me
I made some weird decisions e.g. to host minio - which seems like another dead end - I don’t recall if Cloudflare R2 wasn’t a thing back then but I gladly moved few GBs of data I own there
coolify is not that cool; just use k3s