SSH OVER AWS SYSTEMS MANAGER: DITCHING KEY PAIRS FOR IAM AUTHENTICATION

If you’ve ever managed SSH keys for EC2 instances at scale, you know the pain. Keys get lost, rotated, shared inappropriately, or worse - committed to Git repos. What if I told you that you could use all your familiar SSH tooling - ssh, scp, rsync, port forwarding, and more - without ever touching an SSH key pair? Enter AWS Systems Manager Session Manager as an SSH proxy.

Why SSH Over SSM?

The traditional SSH model requires:

SNAPSHOT TESTING WITH SYRUPY IN AWS CDK PYTHON

In 2023, I wrote a post on snapshot testing in AWS CDK Python. Since then, I’ve switched from pytest-snapshot to syrupy.

At Defiance Digital, we use the AWS CDK for almost everything. Generally we use TypeScript because it’s the original language for the CDK, everything using JSII transpiles back to TypeScript, and it has the most compability with the CDK. However, we have a few projects that use Python, and on those I’ve really been missing Jest snapshot testing.

GUNICORN IN CONTAINERS

There is a dearth of information on how to run a Flask/Django app on Kubernetes using gunicorn (mostly for good reason!). What information is available is often conflicting and confusing. Based on issues I’ve seen with my customers at Defiance Digital in the last year or so, I developed a test repository to experiment with different configurations and see which is best.

tl;dr The conventional wisdom to use multiple workers in a containerized instance of Flask/Django/anything that is served with gunicorn is incorrect - you should only use one or two workers per container, otherwise you’re not properly using the resources allocated to your application. Using multiple workers per container also runs the risk of OOM SIGKILLs without logging, making diagnosis of issues much more difficult than it would be otherwise.

PRIVATE DOCKER REGISTRY IN K8S

Mirroring package repositories has been an option available to Linux users for a long time. It’s a great way to save bandwidth and speed up package installation. This is especially true if you’re using Kubernetes, where you’ll be pulling images from a registry many times a day. There’s a lot of value to doing the same with Docker images, particularly for any that are private and only in active use in your homelab.

SNAPSHOT TESTING IN AWS CDK PYTHON

NOTE: Since writing this post, I’ve switched from pytest-snapshot to syrupy. Check out my new post for more information.

At Defiance Digital, we use the AWS CDK for almost everything. Generally we use TypeScript because it’s the original language for the CDK, everything using JSII transpiles back to TypeScript, and it has the most compability with the CDK. However, we have a few projects that use Python, and on those I’ve really been missing Jest snapshot testing.

NOMAD HOMELAB

In my ongoing quest to become a mad scientist, I realized I had a bunch of compute sitting around my home, largely unused. I could try to choose what runs on which machine and track and manage it all…

So instead, I embarked on a quest to make it all part of one big Nomad cluster at home. The Nomad server is a single qemu VM running on my Synology NAS system, which is also on a UPS to keep it running until my generator kicks on. Since Hashicorp Nomad separates the notion of server (orchestrator) and client (worker), that single VM is all I really need to schedule basically any workloads I need to at home.

INSTALLING HASHICORP NOMAD CLIENT ON A MAC - PERSISTENT

I’ve been setting up Hashicorp Nomad as my scheduler for my homelab, which includes numerous Raspberry Pis, a few Linux VMs, an Ubuntu laptop, and several Macs. Getting the server set up was fairly straightforward - I’m taking an iterative approach with a single bootstrapped server for now - but I had trouble finding much in the way of setting up persistent clients.

This blog is me leaving breadcrumbs for my future self and any other folks who want to take advantage of the power of Hashicorp Nomad clients on Macs in their own homelabs.

THE CLOUD RESUME CHALLENGE - AWS

The Challenge

Ever since Forrest Brazeal released his original Cloud Resume Challenge, I’ve been curious to take it for a spin. As a former educator who pivoted into a career in technology, with the vast majority of it in cloud, it hit all the sweet spots for me. Clear, concise, project-based, relevant, and challenging enough to demonstrate all the competencies that a cloud engineer would need to be successful.

I finally sat down to do the challenge myself. I’ve been in this field for a while and had all the theoretical knowledge, plus I’d done much of it in various work projects before. This should be easy, right?