Obviously not me directly, I just had my clankers do it. Did I learn anything? Not really. Was it worth it? Probably not.

But first, some history.

The app I’m referring to is almost-dead, mostly visited by bots and spammers, it’s a service about Quake. It has some old content, some forums, and once every few months somebody will post a ‘hey guys, are you still alive?’ comment which will get 1 response after maybe a few weeks. It’s fascinating to observe. I even got sentimental and wrote some news after the migration.

I still keep it because it was my first ‘real’ web app written in Ruby on Rails. Plus I wasted spent my youth there as a kid arguing with strangers on the forums, posting stuff and whatnot. It’s mostly a CRUD, but unfortunately when you look closer you will see a myriad of functionalities buried underneath - permissions with roles, versioning, moderation panels, preferences, in-app notifications, file upload and file post-processing, a few background workers etc. Mostly basic stuff - but it does add up in terms of lines of code.

Obviously I would never rewrite it myself - I had that idea a few times, but given the amount of effort - why bother? But nowadays, armed with clankers in the form of codex, opencode, claude and a myriad of LLM models I could just do it.

So slowly, over two weeks I did that after hours, because - why not? After all I had some spare allowance to burn tokens in a few subscriptions.

So what are the learnings here?

There are no learnings, really. Figuratively and literally. Because - as said - scope adds up when you look at the app as a whole, and tracking any progress gets mentally taxing pretty quickly.

With such a rewrite you’re probably gonna tell your agents to track progress somewhere, but despite that things will fall through the cracks and I don’t see how you can not correct the course during the process; one wrongly interpreted sentence and agents will start building some really crazy (and totally useless) functionalities or abstractions; then again I didn’t spend weeks reviewing the perfect migration plan.

There is also no learning from a human perspective - because code grows so fast it’s obviously getting impossible to track or build any mental model of it, so it all feels alien and I found myself asking the fundamental question of how things were constructed, which is not a great feeling.

However, it’s amazing such things are even possible nowadays and the app mostly worked - I had to do a few follow-up sessions to fix broken things, do security-related improvements and port things that the llm forgot (gotta love context rot) - but the fact is I would never be able to do it myself given time constraints.

Indeed I feel like software economics will change, unless of course tokens start costing us actual money and we’ll be back to replacing expensive LLMs with cheaper junior developers, lol. At the same time I wonder who’s gonna support all those machine generated lines of code generated at massive scale nowadays.

As a bonus - some useless stats from scc:

scc rails/
───────────────────────────────────────────────────────────────────────────────
Language            Files       Lines    Blanks  Comments       Code Complexity
───────────────────────────────────────────────────────────────────────────────
Ruby                  667      42,902     7,199     7,350     28,353        838
Sass                   29       4,534       533       314      3,687          0
Ruby HTML              28         866       109         0        757         80
YAML                   22       1,619       104        33      1,482          0
Markdown               16       2,663       684         0      1,979          0
JavaScript             15       2,301       160       391      1,750        297
CoffeeScript           11       1,182       128       376        678         64
Plain Text              8         890       155         0        735          0
Gemfile                 7         294        39        91        164          0
Rakefile                7         584        65        85        434         18
HTML                    6         344        46        13        285          0
BASH                    5          24         7         5         12          3
CSS                     4         319        29        27        263          0
JSON                    4      11,837         0         0     11,837          0
Dockerfile              1          68        10         0         58         10
License                 1          21         4         0         17          0
SQL                     1       1,187         3         0      1,184          0
SVG                     1         438         0         0        438          0
Shell                   1          18         2         4         12          2
XML                     1       6,899         0         1      6,898          0
───────────────────────────────────────────────────────────────────────────────
Total                 835      78,990     9,277     8,690     61,023      1,312
───────────────────────────────────────────────────────────────────────────────

scc go/
───────────────────────────────────────────────────────────────────────────────
Language            Files       Lines    Blanks  Comments       Code Complexity
───────────────────────────────────────────────────────────────────────────────
Go                    471     107,544    10,464    12,060     85,020     10,166
HTML                   52         439        54        13        372          0
SQL                    37       5,656       547     1,661      3,448          8
Sass                   31       4,719       543       364      3,812          0
JavaScript             24       3,976       230       560      3,186        412
Markdown               23          41         4         0         37          0
Templ                  22       1,324        69       193      1,062         63
YAML                   11       1,166        35        39      1,092          0
XML                     4          34         0         0         34          0
CSS                     3         243        18        27        198          0
JSON                    2          27         0         0         27          0
Ruby                    2          87        10        29         48          0
Docker ignore           1           6         0         0          6          0
Dockerfile              1          60        14        18         28          1
Plain Text              1          19         4         0         15          0
SVG                     1         438         0         0        438          0
───────────────────────────────────────────────────────────────────────────────
Total                 686     125,779    11,992    14,964     98,823     10,650
───────────────────────────────────────────────────────────────────────────────

Not very interesting numbers due to the sheer difference between stacks (a lot of rails code was just hidden in gems). If you drop sqlc queries/generated code and generated templates, exclude vendored gems and vendored js libs in both apps, some helper bash/ruby scripts and so on - you would see something like:

┌────────────────────┬───────┬────────────┐
│                    │ Files │ Code lines │
├────────────────────┼───────┼────────────┤
│ Go app (go/)       │ 603   │ 74,481     │
├────────────────────┼───────┼────────────┤
│ Rails app (rails/) │ 625   │ 23,778     │
└────────────────────┴───────┴────────────┘

The only problem is that I can somewhat reason behind this Rails code from 2011, but I have mostly no idea what’s going on in the Go port. But who cares, right? If it works, it works?

So the natural next step would be rewrite in Rust, I guess?

I am really happy with this purchase, as it’s been serving me well since 2018, and I hope it will run for few more years. I run ot it myriad of containers, and fairly recently I even tried moving some workloads from Hetzner to a local Ubuntu VM running on DSM. It’s packed with four 4TB WD Red HDDs since I owned, and in general whole unit has been hassle-free.

If you want to max out this unit further, you can consider the following.

1. Adding SSD drives for storage (I was lucky to buy two 2TB Samsung EVO drives before price hike) - you can work around cache-only limitation using this tool. After major DSM upgrade (which you’re gonna do once as this device reached EOL, lol) you might need to do a full reboot as Synology might be beeping and reporting faulty drives. Worry not - you should not lose any data there.
2. Upgrading RAM to 16GB - it seems 2x8GB is the max what Celeron J3455 can handle (according to random sources), but I didn’t verify it personally; full mem test finishes successfully with two 8GB sticks, so I’m pretty happy with that. In case somebody gonna Google that - I bought used Samsung M471B1G73EB0-YK0 and had no problem with it.
3. Getting UPS (d’oh) - I use some old CyberPower where I replaced battery fairly recently - Synology can read UPS status via USB connection, so you can see power status/remaining battery and configure graceful shutdown if needed.
4. Last, but not least - getting some external SSD/HDD for backups of NAS itself. I do backup most precious stuff - like photos and some stuff in home directories - HyperBackup works like a charm.

The only thing i wish this unit had is a little bit of more juice - the CPU is definitely dated; on the other hand that kinda forced me to optimize and trim my workloads, so I’m gonna look at the bright side.

In the time since, I was loosely consulting on a small/medium Rails project that was struggling with some technical debt - one aspect of which was a flaky test suite. The project had 231 flaky test reports, around 6,600 unit tests, and 342 feature specs running on RSpec, Capybara, and Selenium-Chrome.

Eventually, it reached 0 (zero) flaky tests - which, I think, makes it the only “production grade” project with such metrics I have ever seen in my life. So, one might ask - how can I get there?

Time

It just takes time. Flaky unit tests are usually easy to reason about; the biggest chunk of work was solving various issues with feature specs, as that area is the Wild Wild West. For example, a rogue request from some container leaking into another test (there was a tusd container for uploads and a test that didn’t wait properly for the upload to finish, meaning a webhook would fail a subsequent, random test). Other issues included custom smooth-scroll logic causing clicks on unintended elements, or JavaScript components not being mounted before an interaction with the page occurred - and the list goes on.

Trying a different driver

I tried using Ferrum (with the Cuprite driver). It won’t auto-magically speed up your test suite, but using a different driver might surface different problems, as Ferrum speaks directly to the browser over a WebSocket (a huge gain since you don’t have to install the right version of chromedriver).

For example, it turned out a lot of Stimulus-related tests now failed because the test tried to click on a button before the Stimulus controller managed to connect. In that case, I would suggest rendering such a problematic button as disabled and re-enabling it in the connect() method. In practice, this shouldn’t be visible to the end-user, and Capybara will wait for the button to be enabled when doing click_on. That simple change fixes your test without any weird wait-for/sleep hacks.

Speaking of which - if you’re explicitly waiting for some elements or interactions outside of Capybara’s default wait behavior, there is a high chance you’re doing something wrong, or maybe your UI lacks a UX interaction you can assert on.

In the end, the project stayed on Ferrum, as it also solved Selenium “invalid session id” exceptions, which turned out to be really difficult to debug in a CI environment.

Logging

Some feature specs can feel impossible to debug. Ferrum offers really in-depth logs that you can shove into your favorite LLM for cross-referencing. In some cases, it can work wonders. Capybara::Cuprite::Driver accepts a logger, so you can save these as CI artifacts in case a test fails. Then just tell your agent to analyze it and you might just get lucky.

Persistence

When you have 100+ flaky tests, the whole thing seems overwhelming. See point 1 and remember - it will take time. Treat it as a marathon, take the easy wins, fix one test a week, and you will be done after 2 years. Having no flaky specs feels really good, and in case something new surfaces, the resistance to addressing it is much smaller.

I know some companies just love heroic efforts, but solving this isn’t about a single weekend of slopfixing - it’s just work, sometimes a bit tedious one.

After Hetzner rolled out their new cloud offering, it seems like AMD on Hetzner is kinda dead end at this point (don’t quote me on this). So I ended up in a weird situation where I had some old dangling ARM k3s node, some new PoC Coolify ARM-based server and it was time to put it all back together - obviously by setting up a third new project.

What went right

• having argocd + kustomize manifests + some ansible sprinkles for initial bootstrap allowed me to quickly re-create foundation of the infrastructure; once I had that, re-LLM-ing refreshed stack with ArgoCD in mind was fairly pleasant (as I only had to scream at the computer to fix those yamls instead of doing it myself)

• I had to refresh one of my app as it was full of security holes, as usual best practices like cough test suite and whole CI/CD pipeline (which needed a nudge) made my life much easier

• streamlining my other clusters prior to the move - I didn’t have to think how to spin up basic stuff like redis-es or postgres-es

• terraform for, you know, avoiding clickops

What went wrong

• general software rot; surprise, surprise - not maintained stuff becomes a tangled mess; expired tokens, outdated keys, broken authorizations, dependencies that no longer exist; the resistance to even start thinking about it is huge

• amount of dependencies, forked dependencies (what was I thinking 12 years ago?!), frontend dependencies in particular (I will live with Bootstrap 2 till the day I die); some hard to update/migrate software - e.g. had to give up on plausible, didn’t want to deal with 2 -> 3 upgrade, clickhouse - I just scraped everything and moved to umami

• terraform; as I had like 3 different repos which managed different parts of infra, and everything eventually got out of date (went through local state, atlantis to terraform cloud route) - but that’s on me

• I made some weird decisions e.g. to host minio - which seems like another dead end - I don’t recall if Cloudflare R2 wasn’t a thing back then but I gladly moved few GBs of data I own there

• coolify is not that cool; just use k3s

So, I have a fairly easy CircleCI configuration for my Rails app. There are a few jobs running on different containers that prepare the dependencies cache, followed by some linters. Then, there are a few parallelized jobs that run tests, a fan-in job that collects test coverage results, and uploads them to Codecov. There’s also an optional step that builds a Docker image, pushes it to ECR, and optionally triggers ArgoCD deployments.

It’s not much, but at the same time, it mimics a setup that you might call a “production” step. There’s caching involved, an automated repository that maintains a CI-relevant Docker image used by CircleCI, and overall, the “waste” in that pipeline is really minimal. CircleCI can also cache those containers on its end (depending on which VM I’ll hit), which is also a nice addition. And Docker layer caching seems to mostly work.

I kind of like CircleCI as a product, but it has those periods where barely anything works; jobs hanging, never finishing. So, I decided to take a peek at the other side.

Getting GitHub Actions “production ready”

Let’s recap what I need from the CI:

• Aggressive caching of dependencies from different ecosystems
• Docker layer caching for the Docker build step
• The ability to parallelize a single job
• (Nice to have) Test timing so I can distribute the tests based on historical execution time evenly

Spoiler alert

In the end, I abandoned this experiment. In the meantime, GitHub announced changes in the pricing model for self-hosted runners (just to postpone it after backslash). I encountered numerous basic issues, such as the inability to provide container arguments, it turned out even if you try to spin up two containers using same image it will fail to start Eventually, the entire setup stopped working because the actions-runner image I was using had become outdated and refused to accept any new jobs (TIL).

Caching

Biggest PITA was caching everything - dependencies and docker layers. I had to hack together:

1. actions-cache-server - which forces you to build your own runner image (but thankfully this whole even exists in the first place)

2. Customization of Docker images: you’ll likely need to customize your existing Docker images to use the runner (1001) user and /home/runner work directory. Otherwise all hell breaks loose due to myriad of assumptions along the way

3. Docker Layer Caching: to cache Docker layers, you’ll need to spin up your Docker registry and make some crazy hacks within the action to force its usage. Here’s an example of how you can set up Docker Buildx:

# NOTE: This spins up a new deployment in Kubernetes and you might need to patch the RBAC provided by ARC.
- name: Set up Docker Buildx
	uses: docker/setup-buildx-action@v3
	with:
	  driver: kubernetes
    # my specific node selector and tolerations so pods land on CI nodes
	  driver-opts: |
		"nodeselector=ci=true","tolerations=key=ci,value=true,effect=NoSchedule"
	  # I spin up basic registry, no auth, access from within cluster only
    buildkitd-config-inline: |
		[registry."docker-registry.docker-registry.svc.cluster.local:5000"]
		  http = true

  # ....

  - name: Build and push
	uses: docker/build-push-action@v6
	with:
	  context: .
	  file: Dockerfile
	  push: true
	  tags: ${{ steps.meta.outputs.tags }}
	  cache-from: type=registry,ref=docker-registry.docker-registry.svc.cluster.local:5000/my-app:latest
	  cache-to: type=registry,ref=docker-registry.docker-registry.svc.cluster.local:5000/my-app:latest,mode=max
	  provenance: false
	env:
	  DOCKER_BUILD_SUMMARY: false
	  DOCKER_BUILD_RECORD_UPLOAD: false

Test Balancing

For splitting test execution based on timing, I found the best solution I could find: split-test. I just preinstalled it on my custom runner image. Assuming you’ll keep timings only on the main branch, it should work pretty well.

Parallelization

In general github actions provide out of the box support for job matrixes which works well. The ability to split CI config into smaller chunks is also a nice addition (along with reusable workflows syntax which is pretty neat once you wrap your head around it).

Conclusion

As I like the Github’s UI around actions and general dev experience when it comes to setting up simple workflows - self hosting it, trying to understand rather not ideal runners source code and all the quirks along the way was not fun. Next I will look into woodpecker because I really miss my Drone CI which I used to use a few years back (where all you had to do was setup two docker containers and things used to work 😆)

It’s been a while, and I have decided to look into ‘operator’ solutions, which should be fully compatible with native MySQL implementations (MariaDB was out of the question because of reasons).

If you want something fully MySQL-compatible, you have two major options (I’m not going to mention operators, which seem to be abandonware at this point):

• Percona Server MySQL Operator - which went from technical preview to general availability fairly recently; it can run percona-server-mysql, which should be a drop-in MySQL 8.x replacement
• official MySQL-Operator from Oracle - where you just run official MySQL database (8.x/9.x)

I tested MySQL-Operator on a small testing cluster and decided to move my small workload after a month of battle testing. Migration was supposed to be a quick 20-minute adventure.

Migration gotchas

I wanted to clone data from an existing & running MySQL instance. It turned out ARM won’t be compatible with AMD. You have to run the same patch version of MySQL, but the architecture difference will cause clone init crash, and you will be forced to scrape your cluster and start over once you fix the problem. That really surprised me (prior to battle testing, all my nodes were running on ARM; I did some shuffling in-between). What I had to do is basically reschedule my existing MySQL to an AMD node as my operator was running on AMD CPUs.

Operator doesn’t handle failure upon that initial initDB.clone nicely - once you try to restart it, it will just ignore clone/donor options - the best bet is to scrape the whole configuration (including volumes and any potential configs leftovers - important) and just start over.

Forming a cluster also takes a while - I have a really tiny workload (~12GB of pure data), but it took like a good few minutes for the cluster to establish and for the operator to finally spin up mysql-router pods - as establishing group replication means you have to duplicate that data 3 (or more, depending on your settings) times - for each MySQL pod.

Backup gotchas

MySQL shell dump utility still suffers from a max_execution_time timeout problem during backups, as it seems like there is still no obvious workaround for that.

You can:

• try to set dumpInstance.dumpOptions and, e.g. lower the bytesPerChunk value - beware that the official Helm chart… doesn’t render at all those extra options, despite the fact that it’s respected by the operator 🤦 so you will have to manually patch the InnoDBCluster resource

• just increase/remove the timeout and move the execution timeout option logic to your app or somewhere (which is what I did in the end)

General gotchas

Some stuff is simply broken - e.g. unix socket for monitoring is broken; monitorSpec is not being respected at all - I worked around those problems by using Argocd multiple sources and just overriding whole ServiceMonitor definition and its config. Seems like patches done by some random contributors come to the Oracle internal bug tracker to die, which is a pity and makes me question the future of this whole official product.

Operational gotchas

The operator doesn’t observe changes in podSpec, which is quite frustrating as to change resource allocation, you will be patching your stateful set by hand, which seems like a half-baked way to operate your database instance.

The good stuff

Seems like mysql upgrades seems to be handled well - bumping serverVersion in helm values does a nice rolling upgrade on the cluster. In all fairness mysql was always pretty good at in-place upgrades (I’m looking at you PostgreSQL).

All the heavy lifting related to managing group replication, leader election, setting up mysql-router, S3 backups seems to just work - which was a major selling point for me despite shortycomings mentioned above.

Every now and then I try to look into “simpler” deployment options than my forgotten kubernetes cluster. I eventually abandoned nomad for k3s after running into constant pain from insufficient tooling. Then I forgot about that cluster for like 2 years and I’m waiting for an epic crash which would force me to redo the whole thing again.

Fairly recently I looked into coolify, happily deployed an app via click ops only to realize it doesn’t allow you to even scale a single container. I had to do some weird app-on-app deployment, copy ENVs from one app to another just to spin up a background worker. There is a terraform provider which seems mostly blocked by insufficient APIs exposed by coolify.

On the other hand we have tools like kamal which seems to be a “capistrano on docker” solution (as per their website). I’m quite confused why Basecamp decides to reinvent the wheel here using plain docker containers; they’re even writing their own proxy now?

Then there are a myriad of other solutions like swiftwave, cozystack, dokku or dokploy.

I think the whole deployment/orchestration management is a really difficult problem - that’s why there are maybe so many solutions as in a way it’s also an exciting problem. But IMHO? It’s not a problem worth solving as it was already solved. k3s by stripping all cloud-related irrelevant stuff and baking in the batteries required to run “The Actual Cluster” makes kubernetes really approachable. This tiny distribution* allows you to do easy stuff, but also is flexible enough to adjust it for your own needs without a ton of hacks, duct tape and gum. Because really quickly you’re going to run into basic problems of scaling stuff up and down, cronjobs, secrets/envs management, observability, zero downtime deployments, need for various one-shot jobs, processes, whatever else. That and more is already covered.

The biggest pro and possibly a con is that the kubernetes ecosystem is so goddamn huge - which is overwhelming at first. It’s difficult to decide what’s actually useful and what is not and it just indeed takes time to learn a bit. But that investment is worth it - as there’s pretty much a guarantee that your problem was already solved by someone else. Boring? Maybe. Pragmatic? Definitely.

*Or you know, just pay some cloud/hosting provider to run cluster for you

• using Cloudflare WARP
• using google sign-in via kubelogin

And you want to have some sort of emergency access while you’re away from your iPad.

You can do it using Github Codespaces, but whole process is not well documented, let’s start with some references which were helpful when I glued all this hack together:

• https://github.com/cmj2002/warp-docker/blob/main/entrypoint.sh
• https://community.cloudflare.com/t/how-to-register-into-a-team-with-linux-and-warp-cli/627971/3
• https://developers.cloudflare.com/cloudflare-one/connections/connect-devices/warp/deployment/mdm-deployment/#linux
• https://github.com/spotsnel/cloudflare-client/discussions/1
• https://community.cloudflare.com/t/cloudflare-zero-trust-for-service-token/432243

First what you wanna do is to configure zero trust

You gonna need a service token:

resource "cloudflare_zero_trust_access_service_token" "warp_codespaces_cli" {
  account_id = var.cloudflare_account_id
  name       = "warp-cli-service-token"
  duration   = "forever"
}

A new policy:

resource "cloudflare_zero_trust_access_policy" "warp_cli" {
  account_id       = var.cloudflare_account_id
  name             = "warp-enrollment-cli"
  session_duration = "24h"
  decision         = "non_identity"

  include {
    service_token = [cloudflare_zero_trust_access_service_token.warp_codespaces_cli.id]
  }
}

Moreover - you will need a to link this policy to your existing warp settings:

resource "cloudflare_zero_trust_access_application" "warp" {
  account_id           = var.cloudflare_account_id
  name                 = "Warp Login App"
  domain               = "<your domain>.cloudflareaccess.com/warp"
  app_launcher_visible = false
  type                 = "warp"

  policies = [
    < any existing policies >
    cloudflare_zero_trust_access_policy.warp_cli.id
  ]
}

So your Device enrollment permissions should look like this:

I don’t recall how I created profile settings for the CLI, but I think that was one thing I did manually:

Then you need some stuff in .devcontainer/ directory

devcontainer.json example:

{
    "name": "Development Container",
    "build": {
        "dockerfile": "Dockerfile",
        "context": ".."
    },
    "customizations": {
        "vscode": {
            "extensions": [
                "redhat.ansible",
                "ms-python.python",
                "ms-azuretools.vscode-docker"
            ],
            "settings": {
                "terminal.integrated.profiles.linux": {
                    "bash": {
                        "path": "/bin/bash",
                        "icon": "terminal-bash"
                    }
                },
                "terminal.integrated.defaultProfile.linux": "bash"
            }
        }
    },
    "workspaceFolder": "/workspaces",
    "secrets": {
        "WARP_CLIENT_ID": {
            "description": "WARP client id, provided by organization secret"
        },
        "WARP_CLIENT_SECRET": {
            "description": "WARP client secret, provided by organization secret"
        },
        "OIDC_CLIENT_SECRET": {
            "description": "OIDC client secret, provided by organization secret"
        }
    },
    "capAdd": ["NET_ADMIN"],
    "remoteUser": "vscode"
}

You will want to set secrets in the codespaces settings. Warp id/secret is obtained from warp_codespaces_cli (if created via terraform). You can remove OIDC if you’re not using kubelogin.

Then goes Dockerfile - I used python as I also had ansible in that repo:

FROM mcr.microsoft.com/devcontainers/python:3.13

RUN apt-get update && apt-get install -y --no-install-recommends \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release \
    git \
    iputils-ping \
    iptables \
    && rm -rf /var/lib/apt/lists/*

# Install Cloudflare WARP client
RUN curl https://pkg.cloudflareclient.com/pubkey.gpg | gpg --dearmor -o /usr/share/keyrings/cloudflare-warp-archive-keyring.gpg \
    && echo "deb [signed-by=/usr/share/keyrings/cloudflare-warp-archive-keyring.gpg] https://pkg.cloudflareclient.com/ $(lsb_release -cs) main" \
        > /etc/apt/sources.list.d/cloudflare-client.list \
    && apt-get update && apt-get install -y cloudflare-warp \
    && rm -rf /var/lib/apt/lists/*

# Install kubectl and k9s
ARG KUBECTL_VERSION=v1.33.1
ARG K9S_VERSION=v0.50.9

RUN set -eux; \
    ARCH="$(uname -m)"; \
    if [ "$ARCH" = "x86_64" ]; then \
    KARCH="amd64"; \
    elif [ "$ARCH" = "aarch64" ] || echo "$ARCH" | grep -q '^arm'; then \
    KARCH="arm64"; \
    else \
    echo "Unsupported architecture: $ARCH"; exit 1; \
    fi; \
    curl -LO "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/${KARCH}/kubectl" \
    && install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl \
    && rm kubectl \
    && curl -L "https://github.com/derailed/k9s/releases/download/${K9S_VERSION}/k9s_Linux_${KARCH}.tar.gz" -o k9s.tar.gz \
    && tar -zxvf k9s.tar.gz -C /usr/local/bin k9s \
    && rm k9s.tar.gz

USER vscode

# Install krew and oidc-login plugin under vscode user
ARG KREW_VERSION=v0.4.5

RUN ARCH="$(uname -m)"; \
    if [ "$ARCH" = "x86_64" ]; then \
    KARCH="amd64"; \
    elif [ "$ARCH" = "aarch64" ] || echo "$ARCH" | grep -q '^arm'; then \
    KARCH="arm64"; \
    else \
    echo "Unsupported architecture: $ARCH"; exit 1; \
    fi; \
    curl -L "https://github.com/kubernetes-sigs/krew/releases/download/${KREW_VERSION}/krew-linux_${KARCH}.tar.gz" -o /tmp/krew.tar.gz \
    && tar -zxvf /tmp/krew.tar.gz -C /tmp \
    && /tmp/krew-linux_${KARCH} install krew \
    && rm /tmp/krew.tar.gz \
    && echo 'export PATH="${HOME}/.krew/bin:$PATH"' >> ~/.bashrc \
    && export PATH="${HOME}/.krew/bin:$PATH" \
    && kubectl krew install oidc-login \
    && rm -rf ~/.krew/tmp

That should give you a container with python, kubectl, k9s, Cloudflare WARP and kubelogin ready to go in Codespaces - fire it up and see!

Now what’s left is to glue it all together, I decided to go with a simple bash script which fires all this machinery up:

#!/bin/bash
if [ ! -e /dev/net/tun ]; then
    echo "- TUN device not found, creating..."
    sudo mkdir -p /dev/net
    sudo mknod /dev/net/tun c 10 200
    sudo chmod 600 /dev/net/tun
fi

echo "- Starting dbus service..."
sudo service dbus start

# give dbus some time to start
sleep 3

# Check if envs are set
if [ -z "$WARP_CLIENT_ID" ] || [ -z "$WARP_CLIENT_SECRET" ]; then
    echo "WARP_CLIENT_ID and WARP_CLIENT_SECRET must be set as environment variables."
    exit 1
fi

file_content=$(cat <<EOF
<dict>
  <key>organization</key>
  <string>!!! --- PUT YOUR CLOUDFLARE ORG NAME HERE -- !!!</string>
  <key>auth_client_id</key>
  <string>${WARP_CLIENT_ID}</string>
  <key>auth_client_secret</key>
  <string>${WARP_CLIENT_SECRET}</string>
</dict>
EOF
)
sudo mkdir -p /var/lib/cloudflare-warp
echo "$file_content" | sudo tee /var/lib/cloudflare-warp/mdm.xml > /dev/null

echo "- Starting Cloudflare WARP service (logs at /tmp/cloudflare-warp)..."
sudo warp-svc --accept-tos >/tmp/cloudflare-warp 2>&1 &

# give warp some time to start
echo "- Waiting for Cloudflare WARP to start..."
sleep 5

echo "- Connecting to Cloudflare WARP..."
warp-cli --accept-tos connect

if [ -z "$OIDC_CLIENT_SECRET" ]; then
    echo "OIDC_CLIENT_SECRET must be set as an environment variable."
    exit 1
fi

# I keep kube-config in my repo, reference below
echo "- Configuring kube config..."
mkdir -p ~/.kube
envsubst < .devcontainer/kube-config.yaml > ~/.kube/config

echo "All set! You can now access your Kubernetes cluster using k9s."

Run that while in codespace’s terminal and that will configure Cloudflare WARP and now you should be able to access your internal network from codespaces container which is pretty neat-o.

For reference my kube-config.yaml (truncated)

# ...
clusters:
- cluster:
    certificate-authority-data: Redacted
    server: https://<internal network ip>:6443
  name: my-cluster
# ...
users:
- name: my-oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://accounts.google.com
      - --oidc-extra-scope=email
      - --oidc-client-id=<<MY CLIENT ID>>
      - --oidc-client-secret=${OIDC_CLIENT_SECRET}
      - --grant-type=device-code
      command: kubectl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

Remember if you want to use device-code you need to configure oauth for limited input - as there is no way to open a browser while in codespaces container, but PIN login still should work nicely.

We - and by we - I mean me with a group of random internet folks - just released something called “Polish Defrag MAPpack no.1” - 9 maps made by 9 different folks. I met those randos online on an internet forum located under q3arena.gry.wp.pl - a site which was moved around between domains since then, but that’s not the point.

Then we did that again - in March 2004, we released part 2 - 9 maps from 9 different folks.

And the last time we did it in October 2005 (took us more than a year to wrap it up) - again, 9 different folks released 9 different maps. At this point, I don’t recall if the “nine” was a coincidence or if we had some hard limit of how many folks could contribute. Probably the first one.

In total, 16 people contributed to 27 maps over a span of 2 years. Those are the nicknames of those randos which I have never met in real life (and the number of maps created by them):

• Pea_IMPOSTOR x2
• ImOoR x1
• gorat x1
• Progner x1
• RuSHeR x2
• T4m8uryn0 x3
• vic x1
• dAde x2
• NemiX x1
• SzakaL x2
• Tek x2
• TymoN x2
• Yacek x2
• Marko x1
• DyZi0 x1
• and me x3

It all happened when I spoke with T4m8uryn0, gave him an idea, he started a single thread on the forum, and then it took off on its own. I never thought about this much.

It was 2003, and yet we managed to collaborate fully async on the private forum on this project - I think we moved to a private forum eventually; it seems it is still online and holds 527 posts and 19 topics - I think we had some huge single public discussion before that. On top, we used IRC from time to time for some real-time interactions. We provided feedback, ran tests on early versions of the maps, created a unified visual identity of map screenshots, and eventually managed to ship something in a self-organized manner. All that twenty-two freaking years ago with a total of zero video calls - I didn’t even knew how those people looked in real life.

In a perspective I find it kinda amazing - we were teenagers, sometimes still kids - and it was - in fact - my very first exposure to a remote work (or rather collaboration) - I just didn’t know it was called like that.

As said - I never managed to meet with any of those folks; we were spread out across whole Poland, a random group of geeks who loved to play very weird Quake 3 modification who decided to make some maps. It’s a pity that Quake was losing its playerbase and the whole forum went dead not so long after. And now I wonder how those people are doing.

The majority of the content from that era is gone and lost forever, I still host quake.net.pl (which I eventually rewrote from PHP as my first Ruby on Rails project). There are some videos on YT but for some reason finding it is extremely hard - thus I created a playlist for anyone who was part of the scene who wishes to take a trip back memory lane.

There is no session context - thus form_tag won’t render the “authenticity token” hidden field - it will silently omit it. This behavior was changed around 2021, and you can check out this lengthy discussion about the problem.

Rails Turbo will automagically attach the CSRF token to the request obtained from a meta tag:

prepareRequest(request) {
    if (!request.isSafe) {
      const token = getCookieValue(getMetaContent("csrf-param")) || getMetaContent("csrf-token");
      if (token) {
        request.headers["X-CSRF-Token"] = token;
      }
    }
    if (this.requestAcceptsTurboStreamResponse(request)) {
      request.acceptResponseType(StreamMessage.contentType);
    }
  }

So if you render a form via Turbo which you intend to submit further without Turbo, you will need to re-attach the CSRF token in a similar manner. You can use a Stimulus controller as shown in the example below:

import { Controller } from "@hotwired/stimulus";
 import Rails from "@rails/ujs";
 
 export default class extends Controller {
   private csrfParam = Rails.csrfParam();
 
   connect() {
     let form = this.element;
 
     // assuming page can be refreshed by hard refresh and form will by
     // rendered with the session context - thus with csrf token
     if (form.querySelector(`input[name='${this.csrfParam}']`) === null) {
       form.appendChild(this.authenticityTokenInput);
     }
   }
 
   get authenticityTokenInput() {
     const input = document.createElement("input");
 
     input.type = "hidden";
     input.name = this.csrfParam;
     input.autocomplete = "off";
     input.value = Rails.csrfToken();
 
     return input;
   }
 }

Then you attach this controller to your form via data-controller (the Stimulus way).

Nameserver limits were exceeded, some nameservers have been omitted.

Have you ever run into such errors on your k3s cluster?

The problem was already mentioned in a few places:

• https://vadosware.io/post/sometimes-the-problem-is-dns-on-hetzner/
• https://github.com/syself/cluster-api-provider-hetzner/issues/77

Some folks suggest modifying etc/resolved.conf, but as my node shipped with netplan (and I didn’t want to deal with that, I trust my hosting provider can handle networking), I decided to create a dedicated resolv config for kubelet needs.

It’s mentioned in the official docs

Technically, you just create a file like:

# /etc/resolv.conf.kubelet
nameserver 1.1.1.1
nameserver 2606:4700:4700::1111

And configure your pass extra kublet arg to your k3s agent:

--resolv-conf=/etc/resolv.conf.kubelet

And that should solve this quite annoying warning message as now you will have two nameservers entries with is within kubernetes limits.

Nowadays (2024), self-hosting Sentry requires spinning up 50+ different services - and that is of course without any fancy HA setup. It’s still doable, especially with Kubernetes, but the learning curve is definitely more steep. Then again, the feature set of Sentry itself is much richer - it’s not about just catching errors anymore; you have full-fledged build-in performance monitoring, session recording, and tons of other observation-related goodies.

One thing is kinda tricky is monitoring of your Sentry instance. Here is how I do it.

Use built-in endpoints

• /_health/ - to ensure your web is working
• /api/relay/healthcheck/live/ - to ensure your Relay is alive
• /api/relay/healthcheck/ready/ - to ensure your Relay is ready for events ingestion

…and it’s not enough

I ran into a situation where Relay went into some reconnecting spree (despite being live/ready) - at the moment I don’t remember the exact root cause, but it took me a while to realize that events are not being properly ingested. In the end, I decided to configure a cronjob which would trigger a Sentry event (exception) to one of my projects every ~hour. Then I muted that exception as obviously it was not actionable.

Once you have that exception in Sentry, it’s time to check if it’s being registered every hour.

Sentry exposes API endpoint where you can obtain issue details; one interesting part in that JSON response is lastEvent - you can query this endpoint every ~hour, parse the response, ensure it’s no older than ~2h (so you can have some overlap), and if it’s not - meaning that Sentry is nicely ingesting events.

You can do that using Cloudflare Workers, here is a part of Terraform code which I used to configure it:

resource "cloudflare_worker_script" "sentry_monitoring" {  
  account_id = var.cloudflare_account_id  
  name       = "sentry-monitoring"  
  content    = file("sentry.js")  
  
  plain_text_binding {  
    name = "ISSUE_ID"  
    text = "<your issue id>"  
  }  
  
  plain_text_binding {  
    name = "SENTRY_DOMAIN"  
    text = cloudflare_record.sentry.name // assuming you have this configured
  }  
}  
  
resource "cloudflare_worker_secret" "sentry_auth_token" {  
  account_id  = var.cloudflare_account_id  
  script_name = cloudflare_worker_script.sentry_monitoring.name  
  name        = "SENTRY_API_TOKEN" // issues read-only token 
  secret_text = var.sentry_api_token // I'm using Terraform Cloud to set this variable
}  
  
resource "cloudflare_worker_route" "sentry_monitoring" {  
  zone_id     = cloudflare_zone.your_zone_id.id // ssuming you have this configured
  script_name = cloudflare_worker_script.sentry_monitoring.name  
  pattern     = "https://${cloudflare_record.sentry.name}/uptime*"  
}

Here is the worker JS file:

addEventListener('fetch', event => {
  event.respondWith(handleRequest())
})

async function handleRequest() {
  // you might want to pass project id as argument here
  const apiUrl = `https://${SENTRY_DOMAIN}/api/0/issues/${ISSUE_ID}/`
  const token = SENTRY_API_TOKEN;

  try {
    const response = await fetch(apiUrl, {
      headers: {
        'Authorization': `Bearer ${token}`
      }
    })
    const data = await response.json()

    if (!response.ok) {
     // worker logs is currently in beta, you can enable it in Cloudflare dashboard,
     // seems like terraform provider is lagging behind
     console.log(`Error: ${response.status} - ${response.statusText}`);
     return new Response('API Error', { status: 500 });
   }

    const lastEventStr = data.lastSeen
    const lastEventDate = new Date(lastEventStr)
    const currentTime = new Date()
    const diffInHours = (currentTime - lastEventDate) / (1000 * 60 * 60)

    if (diffInHours <= 2) {
      return new Response(lastEventStr, { status: 200 })
    } else {
      return new Response(lastEventStr, { status: 500 })
    }
  } catch (error) {
    console.log('Error:', error);
    return new Response("Unhandled error", { status: 500 })
  }
}

Then under <your Sentry domain>/uptime I would have a Cloudflare worker returning 200/500 response - now it’s easy enough to add that endpoint as well to monitoring software of your choice.

Now you should be monitoring 4 different endpoints which should give you enough confidence that Sentry is indeed up & properly processing events.

As the process is usually quite a chore - you have to snap some pics, write some meaningful description, establish reasonable price, then post it in one or two different places - I thought about automating it a little bit. Seems like LLM APIs are a commodity nowadays I went with vision offered by openai - it turned out single API call with a simple prompt and few pics attached can yield some meaningful results.

Aw shiet, another weekend project it is then.

Iteration, iteration, iteration

I knew I would new need some sort of UI - I started to look into Phoenix - because why not plan with Elixir again? Framework introduced a new concepts (live view) and overall I felt extremely rusty and overwhelmed and made zero progress in an hour or so.

When quickly bootstrapped Rails just to be annoyed with the complexity of the front-end/JS world, also learning another assets pipeline tool felt like waste of time for what I was trying to achieve.

I quickly ditched whole web-app idea and decided that simple CLI would do the job. I thought that it’s a great time to play with bubbletea, but I struggled to structure maintainable multi-step so in the end I laded with Ruby again as tty-prompt covered like 99% of my needs. Ruby again it was then.

Flow

The data flow was relatively simple:

• ask for directory with prepped (properly cropped images)
• ask for optional context about the things I’m trying to sell
• ask which images should be sent to openai for analysis
• resize all images, upload selected ones to Cloudflare R2
• fire it away with a simple prompt, attach images using links to R2

As a result I would get a json response with title, description and suggested price as a response

Getting more price data

The thing with price it that it usually will be off, so I quickly fired up mitmproxy, opened local OLX app on my phone and decoded how their search works to get some more price-related data.

As a follow-up step I prepped a new prompt which exposed a function to item search that was piggybacked on the OLX one—just to have some confirmation on the suggested price.

Posting an ad

I wanted to automate posting ad as well on two different services, but after looking what happens in the mobile OLX API I decided it will be way easier just to automate browser at this point just to fill like 90% of the form. I decided to connect with existing Chrome profile to not have to deal with constant sign-in/captchas which turned out to be more complicated than expected; thankfully someone solved this problem in this pull request; after monkey patching Ferrum I was ready to go. I appreciate that both services were nice enough to have data-testid attributes on elements I needed so scripting whole thing took minutes in the end.

Show me the money

I failed to record proper demo of this thing but I made a poor screenshot for posterity.

The cool thing about it in the example above is that I attached a SMART scan screenshot and told llm to read it’s technical condition (and hoped for the best).

Own your notifications

If team members are pinging you somewhere, ensure it doesn’t go to /dev/null. Streamline your notifications from the apps you’re using. I avoid email like the plague, so I tend to integrate almost everything with Slack - like product management and knowledge base tools. I handle GitHub notifications via a Raycast plugin, so those are nicely grouped per PR/issue and quickly available from my toolbar.

Own your calendar

Set busy slots when you’re, well, busy. Subscribe to calendars worth following (e.g., your PagerDuty schedule or your team’s time off). Moreover, integrating those calendars with your system might help you see the bigger picture.

Own scheduled meetings

Are you scheduling a meeting? Assuming it’s not a regular coffee chat, set the agenda up front. Make notes during the meeting and ensure you leave with some action points. If you receive an invite, accept or reject it - don’t leave it hanging forever. And most importantly, try to be on time.

Own your statuses

Going on PTO? Set your Slack status, set your GitHub status (and mark yourself as busy so you won’t be auto-assigned code reviews). If you’re going to be away for a while, set your status. Leaving for a prolonged lunch break? Status. Have a doctor’s appointment - believe it or not - status. Our best collaboration happens because of statuses.

These small steps can make a big difference in how you’re seen and how well your team runs. So give it a try - your coworkers will love you for it!*

*Results not guaranteed, no refunds

Unfortunately the Cloudflare docs can be quite confusing and sometimes even contradicting - it seems the product evolved quite rapidly and the documentation or blog posts were lagging behind and the old one - were never updated.

Let’s go over common use cases we could use Zero Trust. I will assume that you’re familiar with Terraform and skip irrelevant code bits.

Locking domain behind github auth

1. Add new github auth per official docs
2. I will assume cloudflare account ID to be stored in cloudflare_account_id variable
3. I will assume your_zone zone and some your_domain A record being configured

Start with identity provider.

resource "cloudflare_access_identity_provider" "github_oauth" {
  account_id = var.cloudflare_account_id
  name       = "GitHub"
  type       = "github"
  config {
    client_id     = "<your GH app client id>"
    client_secret = "<your GH app secret>"
  }
}

Then grant access to your GH organization.

resource "cloudflare_access_group" "github_org" {
  account_id = var.cloudflare_account_id
  name       = "Github Organization Group"

  include {
    github {
      identity_provider_id = cloudflare_access_identity_provider.github_oauth.id
      name                 = "<name of your github org>"
    }
  }
}

And glue it together by protecting your domain within the zone.

resource "cloudflare_access_application" "example" {
  zone_id                   = cloudflare_zone.your_zone.id
  name                      = cloudflare_record.your_domain.name
  domain                    = cloudflare_record.your_domain.name
  type                      = "self_hosted"
  session_duration          = "24h"
  auto_redirect_to_identity = true
  allowed_idps = [
    cloudflare_access_identity_provider.github_oauth.id
  ]
  policies = [
    cloudflare_access_policy.example.id,
  ]
}

resource "cloudflare_access_policy" "example" {
  account_id       = var.cloudflare_account_id
  name             = "github auth"
  session_duration = "24h"
  decision         = "allow"

  include {
    group = [
      cloudflare_access_group.github_org.id
    ]
  }
}

Locking specific path

It’s exactly the same as the above, but you customize the domain bit.

resource "cloudflare_access_application" "example" {
  zone_id                   = cloudflare_zone.your_zone.id
  name                      = cloudflare_record.your_domain.name
  domain                    = "${cloudflare_record.your_domain.name}/some-path"
  type                      = "self_hosted"
  session_duration          = "24h"
  auto_redirect_to_identity = true
  allowed_idps = [
    cloudflare_access_identity_provider.github_oauth.id
  ]
  policies = [
    cloudflare_access_policy.example.id,
  ]
}

Using WARP client to access internal network

One cool thing you can do is install cloudflared daemon on one of your host in the internal network and create a secure tunnel between that network and grant access to it using WARP client. There are tons and tons options there, but I will provide most basic example.

I’m not gonna lie that it’s been ages since I set this one up, so my knowledge is a little bit rusty here. Example below allows traffic 10.0.0.0/16 to be routed by the tunnel and accessed when signed into Zero Trust account via WARP client using Github auth (from the example above).

Cloudflared itself can be installed on the host e.g via debian/rpm package. Once installed configure tunnel via cloudflared service install <your tunnel secret> command.

resource "cloudflare_device_settings_policy" "warp_policy" {
  account_id            = var.cloudflare_account_id
  name                  = "WARP settings policy"
  description           = "WARP settings policy"
  default               = true
  enabled               = true
  allow_mode_switch     = true
  allow_updates         = true
  allowed_to_leave      = true
  auto_connect          = 0
  disable_auto_fallback = true
}

resource "cloudflare_split_tunnel" "internal_network" {
  account_id = var.cloudflare_account_id
  policy_id  = cloudflare_device_settings_policy.warp_policy.id
  mode       = "include"
  tunnels {
    address     = "10.0.0.0/16"
    description = "internal network"
  }
}

resource "cloudflare_device_posture_rule" "device_rules" {
  account_id = var.cloudflare_account_id
  name       = "Devices posture rule"
  type       = "warp"
}

data "cloudflare_access_application" "warp" {
  account_id = var.cloudflare_account_id
  domain     = "<cloudflare org name>.cloudflareaccess.com/warp"
}

resource "cloudflare_access_policy" "warp" {
  account_id       = var.cloudflare_account_id
  application_id   = data.cloudflare_access_application.warp.id
  name             = "warp enrollment policy"
  precedence       = "2"
  session_duration = "24h"
  decision         = "allow"

  include {
    group = [cloudflare_access_group.github_org.id]
  }
}

resource "cloudflare_tunnel" "example" {
  account_id = var.cloudflare_account_id
  name       = "<your-server-name>"
  secret     = base64encode("<your tunnel secret>")
}

resource "cloudflare_tunnel_config" "example_config" {
  account_id = var.cloudflare_account_id
  tunnel_id  = cloudflare_tunnel.example.id

  config {
    warp_routing {
      enabled = true
    }
    ingress_rule {
      service = "http://127.0.0.1"
    }
  }
}

resource "cloudflare_tunnel_route" "tunnel_route" {
  account_id = var.cloudflare_account_id
  tunnel_id  = cloudflare_tunnel.example.id
  network    = "10.0.0.0/16"
}

Once connected you can just do ssh 10.0.X.Y from your computer and things should just work (assuming there is a working node in your internal network with that IP that is).

💡 NOTE: Interested in mysql-operator instead? See this blogpost.

Official kubernetes docs have a page about running replicated statefulset using mysql, unfortunately:

• example uses mysql 5.7
• it’s quite hard to grasp - it uses some custom percona-based image and assumes that you might want to scale statefulset into infinity
• as both images are dated it just won’t work on ARM machines

I think in small to medium organisations/teams/projects what you usually want it some finite amount of replicas - maybe one that you could use for some read-only operations and for doing backups from. So let’s try to do that.

Let’s create a two separate statefulsets under same namespace - one will be leader and another one follower - we will implemented GTID-based replication.

First we’re gonna need to prepare xtrabackup docker image:

FROM bitnami/percona-xtrabackup:8.2.0-1

USER root

RUN apt update && \
    apt install -y ncat

The extra bit is ncat which is technically a network cat command. Build it and push it into the wild so your k8s cluster can pull it from somewhere.

docker build . -t my-image/percona-xtrabackup:8.2.0-1

Then let’s prepare a leader statefulset.

Note: I’m going to simplify the yamls below a little bit and omit irrelevant bits

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
spec:
  serviceName: mysql
  selector:
    matchLabels:
      app: mysql
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - name: mysql
          image: mysql:8.2
          args:
            # leader options
            - "--server-id=1"
            - "--gtid-mode=ON"
          env:
            - name: MYSQL_USER
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: MYSQL_USER
            - name: MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: MYSQL_DATABASE
            - name: MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: MYSQL_PASSWORD
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: MYSQL_ROOT_PASSWORD
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - mountPath: /var/lib/mysql
              name: mysql
        - name: xtrabackup
          image: my-image/percona-xtrabackup:8.2.0-1
          ports:
            - name: xtrabackup
              containerPort: 3307
          command:
            - bash
            - "-c"
            - |
              set -ex
              cd /var/lib/mysql

              # Start a server to send backups when requested by peers.
              exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
              "xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root --password=$MYSQL_ROOT_PASSWORD"              
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: MYSQL_ROOT_PASSWORD
          volumeMounts:
            - name: mysql-cloud
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql
          persistentVolumeClaim:
            claimName: mysql

1. We’re gonna explicitly set the server_id and enable GTID based logging in our mysql container.
2. We’re gonna start xtrabackup so follower can request fresh copy of the database

That alone should work - you should have a pod with two containers running - mysql and xtrabackup waiting for the requests.

Now let’s create a follower mysql user using mysql command line:

CREATE USER 'follower'@'%' IDENTIFIED BY '<your password>';
GRANT REPLICATION SLAVE ON *.* TO 'follower'@'%';
FLUSH PRIVILEGES;

and store that REPLICATION_USER and REPLICATION_PASSWORD in mysql-follower secret - we’re going to need that in the follower.

Speaking of - the follower bit:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-follower
spec:
  serviceName: mysql-follower
  selector:
    matchLabels:
      app: mysql-follower
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: mysql-follower
    spec:
      initContainers:
        - name: clone-mysql
          image: my-image/percona-xtrabackup:8.2.0-1
          command:
            - bash
            - "-c"
            - |
              set -ex

              # Skip the clone if data already exists.
              [[ -d /var/lib/mysql/mysql ]] && exit 0

              # Clone data from leader
              ncat --recv-only mysql 3307 | xbstream -x -C /var/lib/mysql
              # Prepare the backup.
              xtrabackup --prepare --target-dir=/var/lib/mysql              
          volumeMounts:
            - name: mysql-follower
              mountPath: /var/lib/mysql
      containers:
        - name: mysql
          image: mysql:8.2
          args:
            # follower options
            - "--server-id=2"
            - "--gtid-mode=ON"
            - "--log-bin=OFF"
            - "--enforce-gtid-consistency=ON"
            - "--super-read-only"
            - "--skip-replica-start=ON" # IMPORTANT: You will want to switch it to OFF after you set everything up
          env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: MYSQL_ROOT_PASSWORD
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - mountPath: /var/lib/mysql
              name: mysql-follower
        - name: replication
          image: mysql:8.2
          command:
            - bash
            - "-c"
            - |
              set -e
              cd /var/lib/mysql

              # This file should be created by init container:
              # https://docs.percona.com/percona-xtrabackup/2.4/xtrabackup_bin/working_with_binary_logs.html
              if [[ -f xtrabackup_binlog_info ]]; then
                echo "xtrabackup_binlog_info exists"
                cat xtrabackup_binlog_info

                [[ $(cat xtrabackup_binlog_info) =~ ^([^.]*.[0-9]+)[[:space:]]+([0-9]+)[[:space:]]+([0-9a-zA-Z:-]+) ]] || exit 1
                rm -f xtrabackup_binlog_info xtrabackup_slave_info

                GTID_MATCHED=${BASH_REMATCH[3]}

                echo "Found GTID match: ${GTID_MATCHED}"
                echo "SET GLOBAL gtid_purged='${GTID_MATCHED}';" > gtid_purged.sql.in
              fi

              # Check if we need to complete a clone by starting replication.
              if [[ -f gtid_purged.sql.in ]]; then
                echo "Waiting for mysqld to be ready (accepting connections)"
                until mysql -p"$MYSQL_ROOT_PASSWORD" -h 127.0.0.1 -e "SELECT 1"; do sleep 2; done

                echo "Initializing replication"

                mysql -h 127.0.0.1 \
                      -p"$MYSQL_ROOT_PASSWORD" \
                      -e "RESET MASTER; \
                      $(<gtid_purged.sql.in) \
                      CHANGE REPLICATION SOURCE TO \
                      SOURCE_HOST='mysql', \
                      SOURCE_USER='$REPLICATION_USER', \
                      SOURCE_PASSWORD='$REPLICATION_PASSWORD', \
                      SOURCE_AUTO_POSITION = 1, \
                      SOURCE_CONNECT_RETRY = 30; \
                      START REPLICA;" || exit 1
                # In case of container restart, attempt this at-most-once.
                mv gtid_purged.sql.in gtid_purged.sql.orig
              fi

              echo "All done"
              trap : TERM INT; sleep infinity & wait              
          env:
            - name: REPLICATION_USER
              valueFrom:
                secretKeyRef:
                  name: mysql-follower
                  key: REPLICATION_USER
            - name: REPLICATION_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-follower
                  key: REPLICATION_PASSWORD
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql
                  key: MYSQL_ROOT_PASSWORD
          volumeMounts:
            - name: mysql-follower
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-follower
          persistentVolumeClaim:
            claimName: mysql-follower

This can be hard to grasp at first glance, what is happening here is:

• in initContainers we’re going to try to clone the mysql using percona xtrabackup from the leader (thus the container with the ncat in our leader statefulset)
• xtrabackup will create a file called xtrabackup_binlog_info and based on that file we will find transaction id so replica starts replicating from the correct point in the transaction history and then we will start replication. Thus the skip-replica-start definition - honestly speaking I’m not sure atm if you can get away with setting this to ON straight away as it’s been a while since I set it up - but give it a try and let me know!

If everything went right you should see in follower logs something like:

Replica receiver thread for channel ‘’: connected to source ‘follower@mysql:3306’ with server_uuid=7ee6b0b5-3370-11ee-bd46-be │ │ 0882b78b76, server_id=1. Starting GTID-based replication.

1. Remove ESLint.
2. Install Prettier.
3. Rewrite yourcodebase with npx prettier . --write
4. Commit changes.
5. Update .git-blame-ignore-revs to remove noise from git blame.

/s

As dependabot is not accepting support for new ecosystems, I decided to give Renovate a try, and OH BOY did I missed out on much. In comparison, it makes dependabot looks like toy for little kids. I have been using it for few weeks now and some pros.

• If you’re running on Kubernetes, it’s extremely easy to self-host via the official Helm chart.

• Even if you’re not, you can run it as part of your CI pipeline as a scheduled job.

• It ships with its dashboard concept, which greatly improves visibility regarding outdated dependencies.

• It is highly configurable when it comes to grouping updates - that includes grouping updates across different ecosystems (need to upgrade some Ruby & NPM packages in one go? Checked)

• have great out-of-the-box support for popular ecosystems; updating both orbs and docker images in your CircleCI configuration? Checked

• Have crazy feature called regexp manager, which in practice will allow you to update anything; the downside - it’s regular expression based

Regex matcher - Real-life example

Let’s say you’re using Terraform to manage a project in ArgoCD using the ArgoCD provider.

Renovate supports out of the box helm_release, but the ArgoCD provider defines its own application where you can specify helm attributes.

One way to work around the problem is to shove versions into variable file(s) and annotate those using comment so renovate can parse those.

resource "argocd_application" "kurded" {
  # details ommited
  spec {  
    source {  
      repo_url        = "https://kubereboot.github.io/charts"  
      chart           = "kured"  
      target_revision = var.kured_version  
      helm {  
        release_name = "kured"  
        values = yamlencode({  
          configuration = {  
            rebootDays   = ["su"]  
          }  
        })  
      }  
    }
  }
}

# variables-helm.tf
variable "kured_version" {
  default = "5.3.2" // renovate: dep=kured chart=https://kubereboot.github.io/charts  
}

We’re going to write custom matcher that will lookup variables-helm.tf file(s) and match whole default... // renovate: lines with capture groups

{
  "customManagers": [  
    {
      "customType": "regex",  
      "datasourceTemplate": "helm",  
      "fileMatch": [  
        "(^|/)variables\\-helm\\.tf$"  
      ],  
      "matchStrings": [  
        "default = \"+(?<currentValue>[^'\" ]+)\" +\\/\\/ renovate: dep=(?<depName>[\\a-z\\\\]+)chart=(?<registryUrl>[^ \\n]+)"  
      ]  
    }
  ]
}

The whole notation is quite cryptic at first glance and everything is well explained in Renovate’s official docs, but what we’re really doing here is telling Renovate “hey, this is a helm chart definition - grab name + version + chart url from the string using predefined capture groups”. Note: you might need to adapt depName matcher for you needs as it’s not very greedy.

This is pretty powerful feature as you can adapt Renovate for your own custom needs without writing any “real” code.

The problem

You are using self-hosted jitsi and Chrome 116+ refuses to ask the user for camera/microphone permissions.

The reason

Jitsi up-to-moment-of-writing this uses deprecated iframe permissions. That will be painful if you’re embedding jitsi under a different subdomain for example. If you look into it’s source you will find:

this._frame.allow="camera; microphone; display-capture; autoplay; clipboard-write; hid; screen-wake-lock"

The solution

I don’t have a good one, you can copy-paste the whole external_api.js under a <script> tag and just modify the offensive line accordingly. Something like this should do it:

this._frame.allow="camera *; microphone *; display-capture *; autoplay *; clipboard-write *; hid *; screen-wake-lock *"

You might want to tweak * to something more specific.

Recently, I have been working on a side project where the majority of the data in the app was stored in a single table. That table grew beyond a maintainable point; major pain points are nothing new, and in my case, it boiled down to:

• except single index I was screwed when it comes to querying it; every query becomes gymnastic where I had to iterate over your collection as a straightforward query would probably hang for days in the database

• forget about adding new columns or modifying the structure; if you’re lucky you might run on a recent postgres/mariadb/whatever engine that allows you to add columns in an instant manner (or you are living the NoSQL live and have a different set of problems); given data growth you’re probably running some legacy version and you’re screwed again

• assuming you can schedule a maintenance window it would probably take days and you have to have another table-size-TB free space for potential table rebuild

• operational-wise it’s another rotten egg - think backups/restore pain

So how to deal with it? As usual - it depends; the natural solution would be sharding/partitioning or even a mix of both. Then with partitioning, you might look for native solutions (provided by your RDBMS) or do it on the application level - which also can make sense.

I was working with MariaDB and Rails and in the end, I chose partial sharding (moving majority of the data into brand new DB with MariaDB’s page compression enabled) and partitioning on application level based on a range - I’m not going to go into technical details here as it’s very application specific, but majority of the new data lands in new partitions (or in my case - physical tables).

As a side product of this decision - separating tables physically allowed me to deal with some minor optimizations per table level. The data was 14y old and there were some inconsistencies caused by various bugs introduced (and then fixed) over the years.

How to do it safely?

I have no idea, but this is how I did it:

First I updated rails, d’oh. Rails 6.1 ships with REALLY good support for multiple databases; you don’t have to think about migrations, or managing multiple schemas - everything just works. Kudos to the Rails team for this feature.

Then - no magical solution - I had to wrap every call that touched that table in some abstraction class. That allowed me to put the read/write from old table/new tables behind a global feature flag. I was also able to control read/writes in a controlled manner so the app could continue to run while we worked on the data migration plan.

As the application was relatively simple narrowing down the entry points was relatively easy, but work itself was still painful. Obviously, with sharding - everything that does joins becomes problematic, and up to this day I still have to rethink few problems and how to solve them given the new database structure.

Moving the data was also quite tricky, Ruby quickly crumbled under the amount of data (hello strings allocation) and in the end I had to hack spaghetti code CLI tool in golang; few takes away for future self:

• using SQLite as a local cache layer for speeding up batching over IDs was a nice idea; the cache grew to a few GB which was a little bit too much for Redis, but SQLite took it without a problem

• trying to use Ruby for this particular problem wasn’t the best idea to begin with - I had a good experience with postgres_fwd but most heavy-lifting was done on the DB level there; in the end I re-used some tests written in rspec and had some integration tests in a way for the whole thing

• I wish coded the ability for graceful resume sooner; thankfully I was able to run the process for the given date range (with some limitations)

• I wish that I added simple Slack notifications sooner - were extra useful just to monitor the process as everything was ad-hoc running on a single node in a screen (welp, I got spoiled with the simplicity of “deployment” - or rather a lack of it - when it comes to shipping small golang apps)

Was MariaDBs’s page compression any good?

Yes! We managed to fit all the stuff along with some relationships within ~600GB, so in our case compression ratio was quite pleasing.

If you’re in Europe you might consider Hetzner which once again offers the best bang for your buck; alternatively - if you’re based in the US - Backblaze might be a better fit. Either way, I recommend using rclone which integrates out-of-the-box with S3 and all the protocols you can think of.

Few tips based on my personal experience:

• enable encryption right away, but be aware that you might run into limitations if you have long filenames, like over 143 characters

• observe AWS costs, as S3 can skyrocket, depending on your buckets structure (and of course - bucket size); to make the backup job less API/transfer heavy a lot of folks experiment with fast-list, size-only, no-traverse and max-age; all this depends on how you structure your data and might depend on backup destination, just don’t be surprised about monthly AWS bill.

• if you’re experimenting with rclone on a metal node using crontab you can stream the logs to syslog from your crontab entry for easier debugging (/usr/local/bin/rclone copy -v rest-of-your-command) 2>&1 | logger -t backup-rclone

• technically you can mount even encrypted drive directly to your local system (even on MacOS) - it can be slightly slow if you try to open ridiculously big bucket/directory

For remote-to-local backups with versioning I also use restic which can be paired with rclone, but I recall running into some edge cases and abandoning the idea.

For the simplicity of this example let’s assume you have a single node k3s cluster running which doesn’t have any external load balance in front of it (otherwise you might need a DNS challenge and here we will use HTTP-01).

The official way of installing cert-manager seems to be just kubectl apply -f, which reminds me of piping response from curl directly to bash with a sudo on top. Anyway, I shall talk about ArgoCD in some other blog post. So:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml

Should spin up 3 new pods in cert-manager namespace and create a bunch of other resources.

Now let’s configure ACME HTTP-01 using Terraform.

First, configure let’s encrypt ClusterIssuer - we’re going to use ClusterIssuer and not Issuer so we can issue certificates in any namespace. Is that somehow less secure? I have no idea, but for the sake of let’s just go with it.

resource "kubernetes_manifest" "letsencrypt_issuer" {
  manifest = {
    "apiVersion" = "cert-manager.io/v1"
    "kind"       = "ClusterIssuer"
    "metadata"   = {
      "name" = "letsencrypt-prod"
    }
    "spec" = {
      "acme" = {
        # NOTE: remember to update email here
        "email"               = "your@email"
        "server"              = "https://acme-v02.api.letsencrypt.org/directory"
        "privateKeySecretRef" = {
          "name" = "letsencrypt-prod"
        }
        "solvers" = [
          {
            "http01" = {
              "ingress" = {
                "class" = "traefik"
              }
            }
          }
        ]
      }
    }
  }
}

Applying the plan should create a new resource, but nothing interesting will happen yet. Now let’s try to generate a new certificate; at this point, you should have a domain (see traefik.mydomain.com part) pointing to your k3s node.

resource "kubernetes_manifest" "traefik_cert" {
  manifest = {
    "apiVersion" = "cert-manager.io/v1"
    "kind"       = "Certificate"
    "metadata"   = {
      "name" = "traefik-cert"
      "namespace" = "kube-system"
    }
    "spec" = {
      "secretName" = "traefik-cert"
      "issuerRef"  = {
        "name" = "letsencrypt-prod"
        "kind" = "ClusterIssuer"
      }
      "dnsNames" = [
        # NOTE: remember to change this
        "traefik.mydomain.com"
      ]
    }
  }
}

When you apply the plan, the cert-manager should create a new resource - cert-manager.io/v1/certificaterequests - and will spin a new temporary pod that will serve an HTTP-01 request. Everything should work automagically with the default k3s traefik configuration (as we specified ingress class in the ClusterIssuer definition).

The new certificate should be stored in secrets as kubernetes.io/tls. Now we can expose traefik dashboard to the world.

resource "kubernetes_manifest" "traefik_route" {
  manifest = {
    "apiVersion" = "traefik.containo.us/v1alpha1"
    "kind"       = "IngressRoute"
    "metadata"   = {
      "name"        = "dashboard-ingressroute"
      "namespace"   = "kube-system"
    }
    "spec" = {
      "entryPoints" = [
        "websecure"
      ]
      "routes" = [
        {
          "kind"        = "Rule"
          # NOTE: remember to change the domain
          "match"       = "Host(`traefik.mydomain.com`) && (PathPrefix(`/dashboard`) || PathPrefix(`/api`))"
          "services" = [
            {
              "name" = "api@internal"
              "kind" = "TraefikService"
            }
          ]
        }
      ]
      "tls" = {
        "secretName" = "traefik-cert"
      }
    }
  }
}

After applying plan once you go to traefik.mydomain.com/dashboard you should see good, old traefik web ui dashboard.

Is there a simpler way?

If prefer achieving same result using more automagical approach you can try using annotations - look for cert-manager.io/cluster-issuer.

What about auto https redirect?

You might be tempted to redirect all non-http traffic to https automatically by configuring a traefik helm chart shipped with k3s by setting chart values to something like:

ports:
  web:
    redirectTo: websecure

That seems to be breaking the cert-manager HTTP-01 challenge so I would recommend doing http->https redirect explicitly by specifying redirect middleware. That said allowCrossNamespace might interest you in case you don’t want to define same middleware in every namespace.

…or you can try using DNS01 challenge instead (you can configure two or more separate ClusterIssuers, which what I ended up eventually due to my DNS entries scattered across few providers because some historical reasons).