This blog post is an excellent point of reference on how you can approach the logging aggregation problem while on Nomad. As I’m using Consul, I followed a slightly different approach. I have decided to deploy promtail as a global service on each node and scrape the /var/nomad/alloc/*logs/* directory. Additionally, I slurp systemd-journal to debug potential system services issues as well. I’m not fully convinced that this is the right way. Still, it seems to do the job - even if the given service doesn’t exist on the particular host that promtail is running on - nothing terrible happens - allocation id directory won’t exist on that host, so we will never scrape anything. The most significant upside here is that scrapings logs from new/existing services require adding a service tag - no need to configure extra sidecar service for each job. So it’s a really continent solution.
It’s astonishing how many companies are still struggling with the remote culture. The worst-case scenario is when people responsible for fostering culture don’t see the problem. Communication was always crucial. With the remote setup, it’s even more vital as writing is simply hard and requires practice. Slack already has an excellent article about etiquette - bookmark it to your general channel or add it to the onboarding guide! I want to recap a few points from that list and add a few own pointers.
…or rather: How I choose to backup databases when using Nomad. When I was researching backup options after switching to Nomad, I considered using something like docker-db-backup. I quickly realized one downside of having to remember to align postgres-client (backup container) with the version of the server (database container). And as I was running at that time five different databases (Postgres/MySQL) it was a deal-breaker for me. After more reading, I have decided to write a bash script that would be using Nomad’s raw_exec and cron capabilities.
I still maintain quake.org.pl - a site established around 1998, which was the biggest portal dedicated to Quake 3 Arena on the polish side of the internet. Nowadays, the traffic there is insignificant, but it had a great run and a very vibrant community back in the golden days. I inherited its original PHP codebase like 12-13 years ago, and over 10 years ago, I decided to rewrite it to Rails. So it was my first more significant, serious Rails project - which real users and actual data.
Want the real answer? Do the work. Thank you for coming to my TED talk. Over one year ago, my GF and I had an idea for creating a simple web app that would extract Kindle clippings into some useful formats like Markdown (to store those in Obsidian) or CSV (for further import). Almost a year ago, I registered a domain, set up a landing in one evening, we started some drafts on code together, and then… nothing.
I wrote (very vaguely) about remote work in mid-2014. I need to stand by the first sentence tho - time does fly by fast! What changed in those 7 years? I left a polish startup. Had a middle-life crisis. Bought a motorcycle. Travelled around souther Europe on that motorcycle. Found a place at a small, fully remote company from Switzerland. And never looked back at a office work since 2018.
Official documentation describes the DNS service discovery configuration rather clearly. If you’re running consul as a system-wide service and running on the latest Ubuntu 20 LTS, you might encounter some gotchas as I did. I’m going to list a few for future-me and random internet strangers ;).
So I started to play around with Hasicorp Nomad, and for the time being, I’m done (well, not really). I have built the new infrastructure around Nomad and tooling provided by Hetzner Cloud - cloud servers (as I was already using those anyway), their load balancer, internal networking, firewall functionality, and attachable data volumes. Advertisement: If you would like to play around with Hetzner - get 20EUR for new Hetzner Cloud account via this referral link
Due to a recent infrastructure redo, I had to upgrade a stack that I did not update for around two years. I had MariaDB 10.3.4 and decided to try to upgrade it directly to 10.7.1. The overall process was relatively painless (data volume wasn’t that big), but a few things caught me by surprise.
Over two years ago I wrote about open source alternatives to GA and I was running countly since then. As I’m still redoing my infrastructure using Nomad from Hashicorp after some struggle I realized that I don’t want to move Countly after all. Reason below.