I’ve been try to get more of my home lab behind Authentik and SSO lately, but that raised a question in my mind. What happens when the identity system is the thing that breaks? How do I get into the rest of my lab?
This post looks at the difference between getting Authentik back itself, fixing access to an application tied to it, and getting into infrastructure when Authentik is completely down. I also walk through where I keep local break-glass access and why I would avoid having one shared emergency credential. Also, we look at why it is so important to test these accounts just like we test our backups.
Curious how others handle emergency access in their self-hosted environments without having just another single point of failure.
I’ve started out treating reliability in my home lab as something I need to test. This, like backups, is not something you just want to leave to assumptions. There are 7 checks I recommend making before I really depend on a service. These are pulling Internet access, testing host failure, killing the app itself, watching predictive health metrics, verifying configuration history, tracing application dependencies, and making sure I can rebuild from documentation and Git instead of memory.
The article ties together lessons I have learned from things like Proxmox HA, Keepalived, Docker, DNS, monitoring, Git version control, and application mapping.
I have quite a lot of Docker Compose stacks spread across many different home lab hosts. I realized the difficult part was not rolling out containers. It was keeping them updated over time.
I kept SSHing into individual servers, finding the right Compose directory, pulling new images, and then redeploying my stacks. Then, once they restart, I check whether everything came back healthy.
In this post I walk through how I started using Komodo to automate part of that workflow. With it, I can monitor container image updates, and automate redeployments. It also allows you to import your Compose stacks, and you can use deployments that are stored in your Git repos without having to change your Docker Compose.
I also look at the difference between Poll for Updates, Auto Update, the global update schedule. Also, see how Komodo can pull from Git when you make changes.
Curious how others are managing this once Docker Compose environments go beyond a single host.
I put together a practical two-hour home lab maintenance task list focused on the things I tend to neglect when everything appears to be working: checking updates, verifying backups, cleaning storage, patching, rebooting long-running server. And, also, validating critical services, and documenting what changed.
The main idea is that it doesn't take a lot of time to get rid of a lot of uncertainty in the lab without it being an all-day exercise.
Curious what other people put at the top of their home lab maintenance checklist?
I stumbled onto a project called ANAS and spent some time looking at what it adds to Proxmox VE. The interesting part is that it runs directly on the Proxmox node. It also adds storage management inside the existing PVE web UI instead of making you have a separate appliance or VM running on top of Proxmox.
It can manage ZFS, SMB/NFS shares, iSCSI targets, snapshots, replication, PBS backup/restore, and its own mixed-size Hybrid RAID implementation. Very cool.
I wrote this up from the angle of whether this could replace the common "Proxmox + TrueNAS VM" setup. I don’t think it replaces TrueNAS across the board, but I think the architecture adds a lot of value and is an interesting take on storage in Proxmox moving forward.
I found that a pair of VMs I thought I had that was redundant had quietly ended up on the same Proxmox host. The services were redundant at the application layer, but not at the physical failure-domain layer on my Proxmox hosts.
I wrote up what I found, how VM placement can drift during migrations and maintenance, and how Proxmox HA resource affinity rules can help keep redundant workloads on separate nodes.
The bigger lesson for me was that two copies of a service do not necessarily mean two separate failure domains.
I had always thought of Keepalived mainly as host-level failover with VRRP. While working on containerized DNS HA, I realized Keepalived track scripts can make failover decisions based on the application itself. This is pretty cool functionality. For instance, with DNS you can check to see whether port 53 or a DNS query is actually responding. Check out the full config here, including rise, fall, interval, and the difference between automatic failback and nopreempt.
Proxmox has just announced 24/7 vendor-direct enterprise support. Also, they are opening a new North American subsidiary. I wrote about why I think this matters beyond just longer support hours, especially for organizations that have been evaluating Proxmox as a serious VMware alternative. The bigger shift is that Proxmox is starting to build more of the enterprise support, procurement, and operational structure around the platform. So they are concentrating on adding more than just hypervisor features. Check it out.
I’ve been running Proxmox for quite some time in the home lab, and one thing I’ve learned is that some choices are easy to make on day one. However, they can get super painful to change once your home lab grows. I wrote up seven of the ones I think are worth planning early, including storage, networking, CPU compatibility, clustering, backups, and infrastructure dependencies.
I’ve run GitLab and Gitea in the home lab. I then decided I wanted to see where Forgejo fits in. What stands out to me is how much it includes in terms of features while still feeling lightweight. It has Git hosting, Actions-based CI/CD, runners, package registries, and OCI/container images. I also looked at where it looks different from Gitea, since from the features they are still pretty close.
This post looks at the difference between getting Authentik back itself, fixing access to an application tied to it, and getting into infrastructure when Authentik is completely down. I also walk through where I keep local break-glass access and why I would avoid having one shared emergency credential. Also, we look at why it is so important to test these accounts just like we test our backups.
Curious how others handle emergency access in their self-hosted environments without having just another single point of failure.