I occasionally write blog posts where I talk about software, hardware, or even plants that I'm working on.
Recent Posts
Signal Lost: An upgrade to Linux broke Postgres
Last week, I upgraded my home server from Ubuntu 24.04 LTS (Long Term Support) to Ubuntu 26.04 LTS. I expected smooth sailing, but you and I should know not to expect that, especially if there’s a blog post coming from it.
As soon as I started my Kubernetes cluster, services were failing to start left and right. Cloudnative-pg cluster pods were stuck crashing and restarting forever. Initially, I thought it was some kind of bug in Longhorn (the distributed storage I use for Kubernetes) because I was getting errors when reading/writing data saying “Remote I/O Error”, but the Postgres error was far more impactful.
Continue reading...The Data Machine Must Grow
It finally happened to me. I dared to log into my LinkedIn account and update my job status because I left my job. I dared to update LinkedIn while I was on vacation in Hawaii.
When I tried to login, I found my account locked for security and they just needed a quick peek at my government ID to make sure I’m legit.
Continue reading...Failure to launch - mdadm edition
I’ve been modernizing my OVH-hosted dedicated servers running my various services moving from a CPU from 2015 to a CPU from 2019. While that’s still not that new, they’re budget servers and I only pay $30 USD a month for them. I managed to upgrade 2 out of 3 of them without any issues, making some improvements as I go, including enabling encryption at rest for all my data. However, the last server proved to be a problem.
Normally, I pay for the server, let OVH install a base OS (like Ubuntu Server), then use nixos-anywhere to remotely wipe the machine and install NixOS over top without any issues. But this time, after rebooting, the server would hang trying to mount my RAID array (each server has 2x SSD drives that I RAID1 for performance and fault tolerance.) I couldn’t SSH to remotely unlock my drives because the SSH would immediately close the connection after negotiating an encrypted tunnel.
OVH included an emergency rescue OS that I could reboot into and inspect the drive, which gave me the error message:
nvme0n1p2 does not have a valid v1.2 superblock, not importing!
In this post, I walk through all the steps I went through to diagnose and arrive at a nixos-anywhere bug caused by a Linux kernel change.
Continue reading...