Failure to launch - mdadm edition

I’ve been modernizing my OVH-hosted dedicated servers running my various services moving from a CPU from 2015 to a CPU from 2019. While that’s still not that new, they’re budget servers and I only pay $30 USD a month for them. I managed to upgrade 2 out of 3 of them without any issues, making some improvements as I go, including enabling encryption at rest for all my data. However, the last one proved to be a problem.

Normally, I can pay for the server, let OVH install a base OS (like Ubuntu Server), then with nixos-anywhere I remotely wipe the machine and install NixOS over top without any issues. But this time, after rebooting, the server would hang trying to mount my RAID array (each server has 2x SSD drives that I RAID1 for performance and fault tolerance.) I couldn’t SSH to remotely unlock my drives because the SSH would immediately close the connection after negotiating an encrypted tunnel.

OVH included an emergency rescue OS that I could reboot into and inspect the drive, which gave me the error message:

nvme0n1p2 does not have a valid v1.2 superblock, not importing!

In this post, I walk through all the steps I went through to diagnose and arrive at a Linux kernel bug.

Continue reading...

Comparing changes in a Nix Flake

Nix/NixOS is a declarative language for defining your entire operating system. I use it on my dedicated servers to be able to apply GitOps for the servers. I define my services in a Git repo, everything from what version of packages to use, to what services should be installed, and how they should be installed. Those servers run Kubernetes which is where most of my services live.

Nix is a beast. The language is quite complicated and I wrote about my challenges. While I’ve gotten used to the language, I still don’t consider it intuitive. With that out of the way, my next challenge is that if I run nix flake update, which updates the packages that come from NixPkgs (which is most packages that you install.) Then I don’t really know what’s changing.

Continue reading...

Managing software updates for my Cdk8s home lab with Renovatebot

In my home lab, I use cdk8s which builds on AWS CDK to define my Kubernetes resources infrastructure as code. A lot of people use Helm which uses YAML programming, which I wrote about why I didn’t like, but cdk8s allows me to write it in TypeScript. I can write reusable classes to reduce duplication, like an Ingress construct that handles all the configuration I need. If you want to read more about the pros and cons of Cdk8s, read the post.

In this post, I’m going to show how I use an open-source system, Renovatebot, to keep my home lab up to date.

Continue reading...