I occasionally write blog posts where I talk about software, hardware, or even plants that I'm working on.
Recent Posts
The Data Machine Must Grow
It finally happened to me. I dared to log into my LinkedIn account and update my job status because I left my job. I dared to update LinkedIn while I was on vacation in Hawaii.
When I tried to login, I found my account locked for security and they just needed a quick peek at my government ID to make sure I’m legit.
Continue reading...Failure to launch - mdadm edition
I’ve been modernizing my OVH-hosted dedicated servers running my various services moving from a CPU from 2015 to a CPU from 2019. While that’s still not that new, they’re budget servers and I only pay $30 USD a month for them. I managed to upgrade 2 out of 3 of them without any issues, making some improvements as I go, including enabling encryption at rest for all my data. However, the last server proved to be a problem.
Normally, I pay for the server, let OVH install a base OS (like Ubuntu Server), then use nixos-anywhere to remotely wipe the machine and install NixOS over top without any issues. But this time, after rebooting, the server would hang trying to mount my RAID array (each server has 2x SSD drives that I RAID1 for performance and fault tolerance.) I couldn’t SSH to remotely unlock my drives because the SSH would immediately close the connection after negotiating an encrypted tunnel.
OVH included an emergency rescue OS that I could reboot into and inspect the drive, which gave me the error message:
nvme0n1p2 does not have a valid v1.2 superblock, not importing!
In this post, I walk through all the steps I went through to diagnose and arrive at a nixos-anywhere bug caused by a Linux kernel change.
Continue reading...Comparing changes in a Nix Flake
Nix/NixOS is a declarative language for defining your entire operating system. I use it on my dedicated servers to be able to apply GitOps for the servers. I define my services in a Git repo, everything from what version of packages to use, to what services should be installed, and how they should be installed. Those servers run Kubernetes which is where most of my services live.
Nix is a beast. The language is quite complicated and I wrote about my challenges. While I’ve gotten used to the language, I still don’t consider it intuitive. With that out of the way, my next challenge is that if I run nix flake update, which updates the packages that come from NixPkgs (which is most packages that you install.) Then I don’t really know what’s changing.