JIT SSH Certificates on k3d: Security Without the Friction
I recently encountered a classic "quick fix" shortcut: exposing Kubernetes internal services
directly as LoadBalancers just to enable Remote-SSH sessions for a few internal users.
It’s a shortcut that "works" for a demo, but it's a security nightmare. No identity validation,
static authorized_keys distribution, and a wide-open attack surface. I refused to
ship it.
The Rule: Shipping a "quick fix" that compromises security is just deferred debt with interest.
The Goal: JIT SSH Flow
The better way is to move identity to the front and make SSH access ephemeral.
I built a system called ssh-vm-flow to prove it. The requirements were simple but strict:
- No static keys: Replace
authorized_keyswith short-lived SSH certificates. - OIDC-backed: Use Dex and real OIDC bearer tokens to authorize issuance.
- Unified Ingress: Route both HTTPS and SSH through a single Traefik entrypoint.
The Stack: Why Rust?
I built the core logic in Rust. When you're handling TCP proxies, JWT verification, and cryptographic signing, you want the safety and performance that Rust provides. No GC pauses, no "it probably works" memory safety.
- Rust JIT Gateway: A custom service that acts as both a TCP proxy for SSH and a signing API for certificates.
- Dex + OpenLDAP: The identity provider stack. Real OIDC tokens, not "vibes."
- Traefik: Routing traffic across multiple ports (
443for HTTPS,2222for SSH) through one ingress. - Vault: Secure storage for the user's public keys.
Automation: Making the Secure Path the Easy Path
A secure system is useless if nobody uses it. I built a CLI called jitctl to handle the
handshake. A user only needs to run one command:
jitctl vm connect
Under the hood, this performs the entire dance: OIDC auth via Dex, key registration in Vault,
fetching a signed cert from the gateway, and updating ~/.ssh/config. If it’s not
seamless, users will find a workaround (and usually that workaround is an insecure one).
Operational Checklist
The security properties mostly come from operations. A few items I treat as non-negotiable:
- CA key lifecycle: Rotate with a plan. Private keys are mounted read-only.
- Audit logs: Log every issuance decision. If you can't answer "who accessed what," you're flying blind.
- Fail-closed signing: If the OIDC token is expired or the audience doesn't match, the signer says "no." Period.
The Payoff
Instead of a wide-open LoadBalancer and static keys that live forever, we have a system that validates identity at every step and issues certificates that expire in minutes.
This is how we build geospatial development platforms that actually scale—not by cutting corners, but by automating the right way to do things.
Infrastructure as a product means making the secure path the easiest path.