☸️ Kubernetes Lab Guide

Your box Replica counts, scaling, and the monitoring stack follow this.
Interactive controls unavailable. This page’s script did not load, so the box-size switcher and domain field are hidden. The guide below is the 4 GB build; differences for an 8 GB box are noted in Phase 0 and Phase 6.
Pick your plan above before you start. The guide rewrites itself — commands you shouldn’t run on a 4 GB box disappear rather than sitting there tempting you.
⚠️ Two things you supply: a domain and a VM. Type your domain in the box above and every hostname in this guide rewrites itself — copy the manifests and they’re already correct. Until you do, they read example.com, a reserved documentation domain that will never resolve, shown underlined wherever it appearsnow replaced throughout.
  • drop.example.com → encrypted file transfer
  • secret.example.com → encrypted notes
Both are subdomains of one domain you already control. You need the ability to create plain A records on it — nothing else.

Phase 0 — Get a VM

Sizing

Here’s what the stack actually consumes, so the plan choice isn’t a guess:

ComponentMemory limit
k3s + Traefik + CoreDNS + metrics-server~800 Mi
Redis256 Mi
Public form (optional)64 Mi
Send × 1512 Mi
PrivateBin × 1128 Mi
PrivateBin × 2256 Mi
Prometheus + Grafana + exporters~2.5 Gi
Total~1.7 Gi of 4 Gi — lots of room
Total~4.4 Gi of 8 Gi — very comfortable
General Purpose 4 GB
2 vCPU · 4 GB DDR4 · 50 GB NVMe · $6/mo
  • One replica of each service
  • metrics-server + Traefik access logs for observability
  • Prometheus optional — now borderline, see Phase 6
General Purpose 8 GB
4 vCPU · 8 GB DDR4 · 100 GB NVMe
  • Two replicas of the stateless services
  • Horizontal pod autoscaling
  • Full Prometheus + Grafana stack
  1. Go to EVLBOX VPS
  2. Click All Services in the nav, then click VPS
  3. Choose the plan (highlighted above)
  4. At checkout, apply code TOOBROKE20 for 20% off
  5. After purchase, you’ll land in the VPS Builder:
🖥️ VPS Builder — configure your machine:
• Pick Ubuntu 24.04 as your OS
• Paste your SSH public key (or the panel can generate one for you)
• Set a hostname (e.g. k8s-lab)
• Click Build — most VPS are online within 15 seconds
• You’ll get an email with your IP address and root credentials
  1. SSH in: ssh root@YOUR_VM_IP

Disclosure: the EVLBOX link is an affiliate link and TOOBROKE20 is a referral code — I get a cut if you sign up. The guide works identically on any Ubuntu 24.04 box, or locally with kind create cluster for free.

Base packages + firewall

What you’re installing: ufw is Ubuntu’s friendly wrapper around iptables — it manages the host firewall. jq is a command-line JSON parser you’ll use constantly to read Kubernetes and Traefik output. curl and wget fetch things over HTTP.
apt update && apt upgrade -y
apt install -y curl wget ufw jq
⚠️ Firewall rules are load-bearing here. A default-deny firewall with only 22/80/443 open will break Kubernetes. Flannel (the pod network) tunnels traffic over UDP 8472, and pods talk to each other across the 10.42.0.0/16 and 10.43.0.0/16 ranges. Block those and CoreDNS dies first, then everything that depends on name resolution — which is everything.
# Cluster-internal traffic — REQUIRED, k3s breaks without these
ufw allow from 10.42.0.0/16          # pod CIDR
ufw allow from 10.43.0.0/16          # service CIDR
ufw allow 8472/udp                   # flannel VXLAN overlay

# Public-facing
ufw allow 22/tcp                     # SSH
ufw allow 80/tcp                     # HTTP (also serves the ACME challenge)
ufw allow 443/tcp                    # HTTPS

# Kubernetes API — scope it to you, never the open internet.
ufw allow from YOUR_ADMIN_IP to any port 6443 proto tcp

ufw enable
ufw status verbose
Port 6443 is the Kubernetes API server, protected by a single bearer token. Exposing it publicly means one leaked kubeconfig is total cluster compromise. If your home IP is dynamic, leave 6443 closed entirely and tunnel: ssh -L 6443:127.0.0.1:6443 root@YOUR_VM_IP
You can also develop entirely on your laptop with kind, then deploy to the VPS later. All manifests are identical except the ingress hostnames.

Phase 1 — Install k3s + Helm

What k3s is: a full, certified Kubernetes distribution packaged as a single ~70 MB binary. Rancher stripped out the cloud-provider plugins and legacy alpha features and swapped etcd for SQLite by default, so it runs happily on one small VM instead of needing a three-node control plane. Everything you learn here transfers to any other Kubernetes.
What comes bundled: the installer also gives you Traefik (the ingress controller — the reverse proxy that routes inbound HTTP by hostname and terminates TLS), CoreDNS (in-cluster DNS, so redis.tbtq.svc.cluster.local resolves), metrics-server (CPU/memory sampling that powers kubectl top and autoscaling), local-path-provisioner (turns storage requests into directories on the node’s disk), and ServiceLB (binds ports 80/443 on the host to the cluster).
curl -sfL https://get.k3s.io | sh -

# Verify
kubectl get nodes
# Should show one node, Ready

# Make kubectl work without sudo
mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config
kubectl get pods -A

# Confirm Traefik is running:
kubectl -n kube-system get pods -l app.kubernetes.io/name=traefik

Install the Helm CLI

What Helm is: the package manager for Kubernetes. A “chart” is a bundle of templated manifests plus a values.yaml of knobs, so you install a twenty-resource application with one command instead of hand-writing YAML. k3s ships the Helm controller (which reconciles chart resources inside the cluster) but not the helm binary — you need both.
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash
helm version

Configure Traefik for Let’s Encrypt

What ACME and Let’s Encrypt are: Let’s Encrypt is a free certificate authority. ACME is the protocol it uses to verify you control a domain — Traefik requests a cert, Let’s Encrypt hands back a random token, Traefik serves that token at http://yourdomain/.well-known/acme-challenge/<token> over port 80, and if Let’s Encrypt can fetch it, you get a certificate valid for 90 days. Traefik has this built in, which is why you don’t need cert-manager.

k3s auto-applies any manifest dropped in /var/lib/rancher/k3s/server/manifests/. No restart needed — the Helm controller watches that directory.

⚠️ ACME storage must be persistent, or you’ll get rate-limited into a wall. Traefik writes every issued certificate and its private key into a single acme.json file. The default Traefik chart has persistence disabled, meaning that file lives on ephemeral pod storage and vanishes on restart. Traefik then re-issues everything from scratch — and Let’s Encrypt caps duplicate certificates at 5 per domain per week. A few restarts and you’re locked out for seven days with no recourse. The persistence block below is not optional.
cat > /var/lib/rancher/k3s/server/manifests/traefik-config.yaml << 'EOF'
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--certificatesresolvers.default.acme.email=YOUR_REAL_EMAIL@example.com"
      - "--certificatesresolvers.default.acme.storage=/data/acme.json"
      - "--certificatesresolvers.default.acme.httpchallenge.entrypoint=web"
      # Uncomment while testing — staging certs are untrusted but unlimited:
      # - "--certificatesresolvers.default.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
    persistence:
      enabled: true
      name: data
      size: 128Mi
      storageClass: local-path
      path: /data
    deployment:
      initContainers:
        - name: volume-permissions
          image: busybox:1.36
          command: ["sh", "-c", "touch /data/acme.json && chmod 600 /data/acme.json"]
          volumeMounts:
            - name: data
              mountPath: /data
    ports:
      web:
        exposedPort: 80
        redirectTo:
          port: websecure
      websecure:
        exposedPort: 443
    logs:
      access:
        enabled: true
        format: json
EOF
What the init container is doing: busybox is a tiny image containing basic Unix utilities. An init container runs to completion before the main container starts. Traefik runs as a non-root user, won’t create acme.json itself, and refuses to load it if the file is group- or world-readable. This one-liner creates the file at mode 0600 first. Skip it and persistence appears configured but silently never works.
Replace YOUR_REAL_EMAIL@example.com with a real address — Let’s Encrypt sends expiry warnings there. While you’re still getting DNS right, uncomment the staging caserver line. Staging certificates make browsers complain but have effectively no rate limit, so you can iterate freely. Switch to production once curl comes back 200.
# The config triggers a new helm-install job. Watch the job, not the deployment —
# `rollout status` can return against the old pods before the job even fires.
kubectl -n kube-system get jobs -l helmcharts.helm.cattle.io/chart=traefik -w
# Ctrl-C once it shows 1/1 Completions

kubectl -n kube-system rollout status deployment traefik --timeout=120s

# Confirm the storage volume bound:
kubectl -n kube-system get pvc

# Check logs for ACME registration:
kubectl -n kube-system logs -l app.kubernetes.io/name=traefik --tail=50 | grep -i acme
# Should see: "Registering with ACME server ..."
✓ k3s running. Helm installed. Traefik has durable certificate storage.

Phase 2 — Redis

mkdir -p ~/tbtq/manifests
cd ~/tbtq
No secrets in this build. Neither Send nor PrivateBin takes a credential — the encryption keys they use are generated in the visitor’s browser and never reach the server. So there’s nothing here to keep out of git, which is a nice property to have arrived at by accident. If you later add a service that does need one, encrypt it with SOPS or sealed-secrets rather than committing it.

manifests/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: tbtq

Redis

What Redis is and why it’s here: an in-memory key/value store, and the only datastore this build needs. Send keeps every upload’s metadata in it — expiry timestamp, remaining download count, owner token — while the encrypted blob sits on disk. Send will not start without it. PrivateBin needs no datastore at all.

manifests/redis.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-data
  namespace: tbtq
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: tbtq
spec:
  selector: { app: redis }
  ports:
    - port: 6379
      targetPort: 6379
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: tbtq
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels: { app: redis }
  template:
    metadata:
      labels: { app: redis }
    spec:
      securityContext:
        fsGroup: 999
      containers:
        - name: redis
          image: redis:7-alpine
          args: ["redis-server", "--appendonly", "yes", "--dir", "/data"]
          ports:
            - containerPort: 6379
          volumeMounts:
            - name: data
              mountPath: /data
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 300m, memory: 256Mi }
          readinessProbe:
            exec:
              command: ["redis-cli", "ping"]
            initialDelaySeconds: 3
            periodSeconds: 5
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: redis-data
Why persistence is on. Redis is normally treated as a disposable cache, but here it holds the only copy of Send’s file metadata. Lose it and every live share link 404s while the encrypted blobs sit uselessly on disk. appendonly yes writes an append-only log to the volume. strategy: Recreate makes the old pod release the volume before a new one claims it — a rolling update would deadlock, since the volume can only be attached once.
kubectl apply -f manifests/namespace.yaml
kubectl apply -f manifests/redis.yaml
kubectl -n tbtq wait --for=condition=ready pod -l app=redis --timeout=60s

# Verify
kubectl -n tbtq exec deploy/redis -- redis-cli ping     # → PONG

# Write a key, kill the pod, confirm it survived
kubectl -n tbtq exec deploy/redis -- redis-cli set canary alive
kubectl -n tbtq delete pod -l app=redis
kubectl -n tbtq wait --for=condition=ready pod -l app=redis --timeout=60s
kubectl -n tbtq exec deploy/redis -- redis-cli get canary   # → "alive"
✓ Redis running. Volume bound. Data survives pod deletion.

Phase 3 — Send (Encrypted File Transfer)

What Send is: the community-maintained fork of Mozilla’s discontinued Firefox Send, kept alive by timvisee. Files are encrypted in the browser with AES-256-GCM before upload, and the decryption key is placed in the URL fragment — the part after #, which browsers never transmit to the server. The server therefore only ever holds ciphertext it cannot read.

The flow:

  1. User picks a file in the browser
  2. Browser generates an AES-256-GCM key via the Web Crypto API
  3. File is encrypted client-side
  4. Encrypted blob streams up over a WebSocket
  5. Server returns a URL with the key in the fragment (#key=...)
  6. Recipient opens the link — the browser decrypts locally
  7. One download or expiry → file deleted
⚠️ Send requires Redis. The documentation lists Redis as optional, but that only applies to the development server. The production container needs REDIS_HOST pointing at a live instance or it crashloops on startup. That’s the Redis you deployed in Phase 2.

manifests/send.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: send-data
  namespace: tbtq
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 20Gi
---
apiVersion: v1
kind: Service
metadata:
  name: send
  namespace: tbtq
spec:
  selector: { app: send }
  ports:
    - port: 80
      targetPort: 1443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: send
  namespace: tbtq
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels: { app: send }
  template:
    metadata:
      labels: { app: send }
    spec:
      containers:
        - name: send
          image: registry.gitlab.com/timvisee/send:latest
          ports:
            - containerPort: 1443
          env:
            - name: BASE_URL
              value: https://drop.example.com
            - name: REDIS_HOST
              value: redis.tbtq.svc.cluster.local
            - name: REDIS_PORT
              value: "6379"
            - name: FILE_DIR
              value: /uploads
            - name: MAX_FILE_SIZE
              value: "5368709120"
            - name: MAX_EXPIRE_SECONDS
              value: "86400"
            - name: DEFAULT_DOWNLOADS
              value: "1"
          volumeMounts:
            - name: data
              mountPath: /uploads
          resources:
            requests: { cpu: 100m, memory: 128Mi }
            limits:   { cpu: 500m, memory: 512Mi }
          startupProbe:
            httpGet: { path: /__heartbeat__, port: 1443 }
            periodSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet: { path: /__heartbeat__, port: 1443 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /__lbheartbeat__, port: 1443 }
            periodSeconds: 15
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: send-data
On MAX_FILE_SIZE. 5368709120 is 5 GB, sized against a 20 Gi volume on a disk. Set this above what the disk can hold and one upload fills the node, which triggers a disk-pressure eviction and takes the whole cluster down with it. Raise it deliberately, not aspirationally.
Why one replica. The volume is ReadWriteOnce — it can only attach to one node at a time. Combined with Recreate, rollouts cleanly stop the old pod before starting the new one. If you need Send to scale horizontally, move storage to S3-compatible object storage (MinIO) with the S3_BUCKET variables and drop the volume entirely.
kubectl apply -f manifests/send.yaml
kubectl -n tbtq wait --for=condition=ready pod -l app=send --timeout=120s

kubectl -n tbtq port-forward svc/send 1443:80 &
curl -s http://localhost:1443/__heartbeat__
# → {"status":"ok"}

# Open http://localhost:1443 — upload a file, test the share link.
✓ Send running with Redis. Files encrypted client-side. One-time download enforced.

Phase 4 — PrivateBin (Encrypted Notes)

What PrivateBin is: a zero-knowledge pastebin, forked from ZeroBin in 2016 and maintained since. Text is encrypted in the browser with AES-256-GCM and the key travels in the URL fragment, so the server stores ciphertext it can’t read. Burn-after-reading, expiry timers, password protection, and comment threads are all built in. The nginx-fpm-alpine image bundles nginx and PHP-FPM into one container with no external dependencies — no database, just a directory.

manifests/privatebin.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: privatebin-data
  namespace: tbtq
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: privatebin
  namespace: tbtq
spec:
  selector: { app: privatebin }
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: privatebin
  namespace: tbtq
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels: { app: privatebin }
  template:
    metadata:
      labels: { app: privatebin }
    spec:
      securityContext:
        fsGroup: 82
      containers:
        - name: privatebin
          image: privatebin/nginx-fpm-alpine:stable
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: data
              mountPath: /srv/data
          resources:
            requests: { cpu: 50m, memory: 64Mi }
            limits:   { cpu: 200m, memory: 128Mi }
          livenessProbe:
            httpGet: { path: /, port: 8080 }
            initialDelaySeconds: 5
            periodSeconds: 15
          readinessProbe:
            httpGet: { path: /, port: 8080 }
            initialDelaySeconds: 3
            periodSeconds: 5
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: privatebin-data
What fsGroup: 82 does. PHP-FPM in this image runs as gid 82 (www-data on Alpine), but provisioned volumes arrive owned by root. fsGroup tells the kubelet to change group ownership of the volume at mount time. Without it, saving a paste fails with a permission error that surfaces in the browser as a generic 500 and leaves nothing useful in the pod logs.
kubectl apply -f manifests/privatebin.yaml
kubectl -n tbtq wait --for=condition=ready pod -l app=privatebin --timeout=120s

kubectl -n tbtq port-forward svc/privatebin 8080:80 &
curl -sI http://localhost:8080 | head -3
# → HTTP/1.1 200 OK

# Open http://localhost:8080 — paste text, set "Burn after reading," send,
# then open the URL in incognito to confirm it self-destructs.
✓ PrivateBin running. Zero-knowledge encryption. Burn-after-read works.

Phase 5 — Ingress + DNS

What an Ingress is: a rule that tells the ingress controller “requests for this hostname go to this Service on this port.” Traefik watches for Ingress objects and reconfigures its routing table live. The certresolver annotation additionally tells it to go request a certificate for that hostname.
⚠️ DNS first, Ingress second. HTTP-01 validation requires Let’s Encrypt to reach your hostname on port 80 at the moment the Ingress is created. Apply it before DNS resolves and the challenge fails, after which Traefik backs off with an increasing delay — you’ll sit there wondering why nothing happens.

Swap in your domain

You set your domain at the top of the page, so every manifest you copied from here already contains it. This step is a no-op — run the grep below to confirm, then move on.
If you copied the manifests with the placeholder still in them, replace it everywhere in one pass now — before certificates get requested against a domain you don’t own. Or just set your domain at the top of this page and re-copy.
export DOMAIN=yourdomain.tld          # ← your actual domain

sed -i "s/example\.com/$DOMAIN/g" ~/tbtq/manifests/*.yaml

# Confirm nothing was missed:
grep -rn "example.com" ~/tbtq/manifests/ && echo "STILL PLACEHOLDERS ABOVE" \
  || echo "all replaced"
⚠️ Send bakes its hostname in at startup. Its BASE_URL is what it prints into every share link, so if you already deployed it with the placeholder, the links it generated point at drop.example.com and are dead. Re-apply and restart it:
kubectl apply -f ~/tbtq/manifests/send.yaml
kubectl -n tbtq rollout restart deploy/send
PrivateBin doesn’t care — it builds links from whatever hostname the browser used.

DNS records

drop.example.com   →  A  →  YOUR_VM_IP
secret.example.com →  A  →  YOUR_VM_IP
# Confirm propagation before continuing:
dig +short drop.example.com
dig +short secret.example.com
# All must return YOUR_VM_IP
Plain A records, nothing in front. Whatever registrar holds your domain, these must be ordinary A records resolving straight to your VM — no CDN, no proxy, no forwarding or URL-redirect record type. Traefik terminates TLS itself and answers the ACME challenge on port 80; anything sitting in the path intercepts that challenge and issuance fails. It also breaks Send, whose uploads run over a WebSocket that most proxy layers drop unless explicitly configured.

The practical consequence is that your VM’s IP is public in DNS. That’s normal for this setup — the firewall from Phase 0 and the network policies from Phase 7 are what’s protecting you, along with whatever DDoS filtering your host provides upstream.
If your registrar only offers CNAME or URL forwarding for subdomains, move DNS to a provider that does plain A records for the apex and subdomains. Nothing else in this guide depends on the registrar.

manifests/ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: send-ingress
  namespace: tbtq
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls.certresolver: default
spec:
  ingressClassName: traefik
  rules:
    - host: drop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: send, port: { number: 80 } }
  tls:
    - hosts: [drop.example.com]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: privatebin-ingress
  namespace: tbtq
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.tls.certresolver: default
spec:
  ingressClassName: traefik
  rules:
    - host: secret.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: { name: privatebin, port: { number: 80 } }
  tls:
    - hosts: [secret.example.com]
The tls block has no secretName on purpose. Normally that field names a Secret holding the certificate. With a certresolver, Traefik supplies the cert from its own acme.json instead, and the tls stanza just declares which hostnames to request.
kubectl apply -f manifests/ingress.yaml

# Watch issuance:
kubectl -n kube-system logs -f -l app.kubernetes.io/name=traefik | grep -i acme

# Verify (30–60s per host):
curl -sI https://drop.example.com/__heartbeat__ | head -3   # → HTTP/2 200
curl -sI https://secret.example.com/            | head -3   # → HTTP/2 200

# Confirm the cert is real, not Traefik's built-in placeholder:
echo | openssl s_client -connect drop.example.com:443 -servername drop.example.com 2>/dev/null \
  | openssl x509 -noout -issuer -dates
# Issuer should read Let's Encrypt, not TRAEFIK DEFAULT CERT
✓ Both services on HTTPS with real certificates. Nothing else is exposed. Renewal is automatic and survives restarts.

Phase 6 — Observability + Scaling

Metrics Server

What metrics-server is: a lightweight aggregator that scrapes CPU and memory usage from each node’s kubelet and serves it through the Kubernetes API. It’s what makes kubectl top work and what autoscalers read from. It is not a monitoring system — no history, no alerting, just current values. k3s includes it, so there’s nothing to install.
kubectl top nodes
kubectl -n tbtq top pods
# If empty, wait 30s for the first scrape: sleep 30 && kubectl top nodes

Staying at one replica

On 4 GB, one replica of each service is the right call — you’re using roughly 1.7 Gi of 4, and the spare is doing real work as OS page cache. Adding replicas here doesn’t buy availability (they’d all land on the same node anyway) and does buy memory pressure. Autoscaling and the Prometheus stack are on the 8 GB path; flip the switch at the top of the page if you want to read that section.
If you want redundancy on a small box, the lever is restartPolicy and probes, not replica count. A single pod with a correct readiness probe recovers in seconds. Two pods on one node share a single point of failure and just cost you RAM.

Scale out

What an HPA is: the Horizontal Pod Autoscaler watches a metric (here, CPU as a percentage of the pod’s request) and adds or removes replicas to hold it near a target. It requires resource requests to be set, which they are, and it requires metrics-server, which you have.
# Confirm you have the headroom first:
free -h
kubectl top nodes

kubectl -n tbtq scale deploy/privatebin --replicas=2

kubectl -n tbtq autoscale deployment privatebin --cpu-percent=70 --min=2 --max=3
kubectl -n tbtq get hpa
Send stays at one replica even here. Its ReadWriteOnce volume can’t be shared, so an autoscaler would just generate pods stuck in Pending. Scale Send only after moving it to object storage. Keep the ceilings modest too — 4 vCPU can’t meaningfully serve twenty pods, and an HPA that can outgrow the node just produces Pending replicas and a confusing dashboard.

Access logs

Where these come from: you enabled JSON access logging on Traefik back in Phase 1. Because every request to every service passes through Traefik, one log stream covers the whole cluster — status codes, latency, hostname, and backend, without instrumenting any application.
# All 4xx/5xx across the cluster:
kubectl -n kube-system logs -l app.kubernetes.io/name=traefik --tail=500 \
  | jq -c 'select(.DownstreamStatus >= 400)
           | {t:.StartUTC, host:.RequestHost, path:.RequestPath,
              s:.DownstreamStatus, ms:(.Duration/1000000)}'

# p95 latency by service:
kubectl -n kube-system logs -l app.kubernetes.io/name=traefik --tail=2000 \
  | jq -s 'map(select(.ServiceName)) | group_by(.ServiceName)
           | map({svc:.[0].ServiceName, n:length,
                  p95:(sort_by(.Duration)[(length*0.95|floor)].Duration/1000000)})'

Prometheus + Grafana

What you’re installing: kube-prometheus-stack is an umbrella Helm chart bundling four things. Prometheus is a time-series database that scrapes metrics endpoints on a schedule and stores them with history. Grafana is the dashboard front-end that queries Prometheus. node-exporter runs on the host and exposes OS-level metrics (disk, network, load). kube-state-metrics exposes the state of Kubernetes objects themselves — replica counts, pod phases, deployment conditions. Together they’re the difference between “what is CPU right now” and “what happened at 3 a.m.”
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.adminPassword="$(openssl rand -hex 16)" \
  --set prometheus.prometheusSpec.retention=7d \
  --set prometheus.prometheusSpec.resources.limits.memory=1Gi \
  --set alertmanager.enabled=false

# Retrieve the generated password:
kubectl -n monitoring get secret monitoring-grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; echo

kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80
# Open http://localhost:3000 → admin / (the password above)
Why those flags. Retention is capped at 7 days and Prometheus memory at 1 Gi so it can’t slowly eat the node. Alertmanager is disabled because there’s nowhere for it to page — turn it on when you have a Slack webhook or PagerDuty key to point it at. And never leave the Grafana password as admin; the moment you expose this through an Ingress it’s an open door.
On Prometheus: with only two small services running, a constrained monitoring stack now fits on 4 GB — see the 8 GB view for the install, and use the same resource limits. It was out of the question when this build had a database in it. The Traefik access logs above still answer most of what you’ll actually ask, so try those first.
✓ metrics-server reporting. Access logs queryable as JSON. HPAs sized to the hardware. Grafana up.

Phase 7 — Network Policies

What a NetworkPolicy is: a firewall for pod-to-pod traffic, enforced by the CNI plugin (k3s uses kube-router for this). The model has two properties worth internalizing. First, it’s default-allow until the first policy selects a pod — then that pod becomes default-deny for whichever directions the policy names. Second, traffic must be permitted on both ends: the sender needs an egress allow and the receiver needs an ingress allow. Writing only half is the single most common way to lock yourself out of your own database.

manifests/netpol.yaml

# 1. Default deny, both directions
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: tbtq
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
# 2. DNS egress — UDP and TCP. Large responses fall back to TCP.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: tbtq
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { port: 53, protocol: UDP }
        - { port: 53, protocol: TCP }
---
# 3. Egress from any pod in tbtq to Redis
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-redis-egress
  namespace: tbtq
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector:
            matchLabels: { app: redis }
      ports:
        - { port: 6379, protocol: TCP }
---
# 4. The matching ingress half, on Redis
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-redis-ingress
  namespace: tbtq
spec:
  podSelector:
    matchLabels: { app: redis }
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: {}
      ports:
        - { port: 6379, protocol: TCP }
---
# 5. Ingress from Traefik to the two web apps
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-traefik-ingress
  namespace: tbtq
spec:
  podSelector:
    matchExpressions:
      - { key: app, operator: In, values: [send, privatebin] }
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
Nothing in this namespace needs internet egress. Send talks to Redis and its own disk; PrivateBin talks to its disk and nothing else. No API keys to validate, no geolocation databases to fetch, no webhooks. That’s why there’s no ipBlock rule anywhere in this policy set — a rare and pleasant position to be in, and worth noticing as a property of picking services that do their crypto in the browser.
Since you installed Prometheus: add a policy allowing ingress to the tbtq pods from the monitoring namespace, or every scrape target will show as down.
kubectl apply -f manifests/netpol.yaml

# --- Verify the policies do what you think ---

# Send CAN still reach Redis:
kubectl -n tbtq exec deploy/send -- \
  node -e 'require("net").connect(6379,"redis.tbtq.svc.cluster.local").on("connect",()=>{console.log("OPEN");process.exit(0)}).setTimeout(3000).on("timeout",()=>{console.log("BLOCKED");process.exit(0)}).on("error",()=>{console.log("BLOCKED");process.exit(0)})'
# → OPEN

# Send CANNOT reach PrivateBin:
kubectl -n tbtq exec deploy/send -- \
  node -e 'require("net").connect(80,"privatebin.tbtq.svc.cluster.local").on("connect",()=>{console.log("OPEN");process.exit(0)}).setTimeout(3000).on("timeout",()=>{console.log("BLOCKED");process.exit(0)}).on("error",()=>{console.log("BLOCKED");process.exit(0)})'
# → BLOCKED

# PrivateBin CANNOT reach the internet:
kubectl -n tbtq exec deploy/privatebin -- \
  timeout 3 wget -q -O- https://example.com || echo "BLOCKED (correct)"

# Public traffic unaffected:
curl -sI https://drop.example.com/__heartbeat__ | head -1
# → HTTP/2 200
These tests use node -e rather than wget or curl, because Send’s image is a slim Node base that ships neither — a test that fails because the binary is missing tells you nothing about your firewall.
✓ Pods isolated from each other. Redis path open. Public traffic unaffected. Both directions tested.

🏁 That’s the build

Two encrypted services on your own domain, behind automatic TLS, on a locked-down single-node cluster. Everything from here is optional and independent — take either, both, or neither.
  • Phase 8 (Helm) — packaging practice. Changes nothing about how the stack runs.
  • Phase 9 (cleanup + CI/CD) — operational habits. Worth it only if you intend to keep this running.

Phase 8 — Helm Chart Optional

Skip this unless you want the Helm practice. Your stack already works, and it will keep working untouched. Helm changes how the manifests are packaged, not how anything runs — the pods that come out the other side are identical. It earns its keep when you have a second environment to deploy the same stack into, or when you want a versioned release history you can roll back. For one cluster you maintain by hand, kubectl apply -f manifests/ is a legitimate final answer.

The reason to do it anyway: Helm is on every job description in this space, and this is a low-stakes place to learn what a chart actually is. Note that it requires rebuilding the namespace, so it costs you the data currently in the cluster.
What Helm is doing for you: a chart is your manifests with the environment-specific bits pulled out into values.yaml and replaced by placeholders. Helm renders the templates, applies the result, and records what it applied as a numbered release revision. That record is the real feature — it’s what makes helm rollback possible, and it’s why Helm is strict about which objects it considers its own.
This phase is a rebuild, not an add-on. Everything up to here you applied with kubectl. Helm refuses to take ownership of objects it didn’t create — helm install over them fails with invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name". So the exercise is: tear the namespace down and rebuild it from a chart. That’s a feature of doing this in a lab. If your chart is incomplete, you find out immediately instead of six months later.

Step 1 — Save anything you care about

There is nothing here to back up. Deleting the namespace reclaims the PVCs, but everything on them is deliberately ephemeral: Send’s blobs are one-time downloads on a 24-hour timer, PrivateBin’s pastes burn on read, and Redis holds only the metadata pointing at those. Any live share links people are holding will break — that’s the whole cost. If that matters right now, wait until they’ve expired rather than trying to preserve them.

Step 2 — Scaffold the chart

cd ~/tbtq
helm create tbtq-chart

# The scaffold ships a sample nginx app. Clear it out, keep Chart.yaml.
rm -rf tbtq-chart/templates/*
: > tbtq-chart/values.yaml

tree tbtq-chart
# tbtq-chart/
# ├── Chart.yaml     ← name, version, appVersion
# ├── charts/        ← subcharts, unused here
# ├── templates/     ← now empty; your manifests go here
# └── values.yaml    ← now empty; your knobs go here
Chart.yaml carries two versions and they mean different things. version is the chart’s own version — bump it whenever you change a template. appVersion is the version of the software being deployed, and is informational only. Forgetting to bump version is harmless on helm upgrade from a local directory, but it makes helm history useless for telling revisions apart.

Step 3 — Decide what does not go in the chart

Nothing sensitive to exclude. This build has no secrets, so unlike most charts you can commit the whole thing. Keep the namespace as a plain manifest anyway — letting Helm own the namespace it installs into causes ordering headaches on uninstall.

So the split is:

  • Stays as a plain manifest: namespace.yaml
  • Becomes a template: redis, send, privatebin, both ingresses, the network policies, the cleanup cronjob
Drop the namespace: field from every template’s metadata. Helm sets it from the -n flag, and hardcoding it means the chart can only ever install one place — which defeats most of the point.

Step 4 — The worked example

Here’s Send converted in full — the more interesting of the two, since it has a volume and real configuration. Compare it against manifests/send.yaml from Phase 3; the only differences are the substituted values and the removed namespace.

# tbtq-chart/templates/send.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: send-data
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: {{ .Values.send.storage }}
---
apiVersion: v1
kind: Service
metadata:
  name: send
  labels: { app: send }
spec:
  selector: { app: send }
  ports:
    - port: 80
      targetPort: 1443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: send
  labels: { app: send }
spec:
  replicas: {{ .Values.send.replicas }}
  strategy:
    type: Recreate
  selector:
    matchLabels: { app: send }
  template:
    metadata:
      labels: { app: send }
    spec:
      containers:
        - name: send
          image: "{{ .Values.send.image }}:{{ .Values.send.tag }}"
          ports:
            - containerPort: 1443
          env:
            - name: BASE_URL
              value: "https://{{ .Values.send.domain }}"
            - name: REDIS_HOST
              value: redis.{{ .Release.Namespace }}.svc.cluster.local
            - name: REDIS_PORT
              value: "6379"
            - name: FILE_DIR
              value: /uploads
            - name: MAX_FILE_SIZE
              value: {{ .Values.send.maxFileSize | quote }}
            - name: MAX_EXPIRE_SECONDS
              value: {{ .Values.send.maxExpireSeconds | quote }}
            - name: DEFAULT_DOWNLOADS
              value: {{ .Values.send.defaultDownloads | quote }}
          volumeMounts:
            - name: data
              mountPath: /uploads
          resources:
            {{- toYaml .Values.send.resources | nindent 12 }}
          startupProbe:
            httpGet: { path: /__heartbeat__, port: 1443 }
            periodSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet: { path: /__heartbeat__, port: 1443 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /__lbheartbeat__, port: 1443 }
            periodSeconds: 15
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: send-data
⚠️ Carry the probes across. It’s tempting to templatize only the obvious fields — image, replicas, resources — and drop the rest for brevity. Lose the startupProbe and a slow first boot gets the pod killed by its own liveness probe, which is the failure from Phase 3 all over again, now with Helm in the blast radius to confuse the diagnosis. The whole manifest moves, not the interesting parts of it.
Three template functions in play. | quote wraps a value in quotes — mandatory for env vars, since Kubernetes requires string values and a bare 6 would render as an integer and fail validation. toYaml serializes a whole values block back to YAML. nindent 12 prepends a newline and indents every line by 12 spaces, which is what makes the nested resources block land at the right depth. The leading {{- trims the preceding whitespace so you don’t get a stray blank line.

Step 5 — values.yaml

Every key here is consumed by a template. If you add a value nothing reads, Helm won’t warn you — it’ll just silently do nothing, which is a genuinely annoying half-hour to debug. Redis and PrivateBin are your exercise; convert them the same way.
# tbtq-chart/values.yaml
podCidr: "10.42.0.0/16"

send:
  replicas: 1              # ReadWriteOnce volume — do not raise
  image: registry.gitlab.com/timvisee/send
  tag: latest
  domain: drop.example.com
  maxFileSize: "5368709120"
  maxExpireSeconds: 86400
  defaultDownloads: 1
  storage: 20Gi
  resources:
    requests: { cpu: 100m, memory: 128Mi }
    limits:   { cpu: 500m, memory: 512Mi }

privatebin:
  replicas: 1
  image: privatebin/nginx-fpm-alpine
  tag: stable
  domain: secret.example.com
  storage: 1Gi
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { cpu: 200m, memory: 128Mi }

redis:
  image: redis
  tag: 7-alpine
  storage: 2Gi
# tbtq-chart/values.yaml
podCidr: "10.42.0.0/16"

send:
  replicas: 1              # ReadWriteOnce volume — do not raise
  image: registry.gitlab.com/timvisee/send
  tag: latest
  domain: drop.example.com
  maxFileSize: "5368709120"
  maxExpireSeconds: 86400
  defaultDownloads: 1
  storage: 20Gi
  resources:
    requests: { cpu: 100m, memory: 128Mi }
    limits:   { cpu: 500m, memory: 512Mi }

privatebin:
  replicas: 2
  image: privatebin/nginx-fpm-alpine
  tag: stable
  domain: secret.example.com
  storage: 1Gi
  resources:
    requests: { cpu: 50m, memory: 64Mi }
    limits:   { cpu: 200m, memory: 128Mi }

redis:
  image: redis
  tag: 7-alpine
  storage: 2Gi
PrivateBin scales, Send does not. Send’s ReadWriteOnce volume means a second replica can only ever co-schedule on the same node, and a rolling update would deadlock on the volume. PrivateBin shares that constraint on paper but the risk is lower, since its pastes are small and short-lived — still, keep Recreate on both.

Step 6 — Inspect before you apply

This is the part worth slowing down for. Helm’s real value in a lab is that you can see exactly what it would do before it does it. Three commands, in order of how much they check.
# 1. Syntax and best-practice check — never touches the cluster
helm lint ./tbtq-chart

# 2. Render templates to stdout. This is the single most useful Helm command.
#    If a value is missing, you see <no value> here instead of a broken pod later.
helm template tbtq ./tbtq-chart -n tbtq | less

# Render one file only:
helm template tbtq ./tbtq-chart -n tbtq -s templates/send.yaml

# Catch unrendered values before they reach the cluster:
helm template tbtq ./tbtq-chart -n tbtq | grep -n "<no value>"
# No output = good

# 3. Full server-side validation, still applies nothing
helm install tbtq ./tbtq-chart -n tbtq --dry-run --debug

Step 7 — Tear down and reinstall

kubectl delete namespace tbtq
kubectl get ns tbtq
# → NotFound. Takes 30–60s; PVCs are reclaimed with it.

# Secrets and namespace first — the chart depends on them existing
kubectl apply -f manifests/secrets.yaml

# Now the chart
helm install tbtq ./tbtq-chart -n tbtq

helm list -n tbtq
kubectl -n tbtq get pods -w
Notice there’s no --create-namespace. Your secrets.yaml already declares the namespace, and having both Helm and a manifest claim the same object is the ownership problem again in miniature.

Step 8 — Break it on purpose, then roll back

Why do this: rollback is the feature you’re installing Helm for, and it is the one you least want to first exercise during an actual incident. Break something harmless now while you have a backup and nothing depends on the box.
# Revision 1 is the install. Look at it:
helm history tbtq -n tbtq
helm get values tbtq -n tbtq

# Break it — a tag that doesn't exist
helm upgrade tbtq ./tbtq-chart -n tbtq --set privatebin.tag=this-tag-does-not-exist

helm history tbtq -n tbtq
# Revision 2, status "deployed" — Helm thinks it worked

kubectl -n tbtq get pods -l app=privatebin
# → ImagePullBackOff. The rollout is stuck but the old pod is still serving.

curl -sI https://drop.example.com/__heartbeat__ | head -1
# → HTTP/2 200  — the deployment's maxUnavailable kept the old pod alive

# Roll back to revision 1
helm rollback tbtq 1 -n tbtq
helm history tbtq -n tbtq
# Revision 3, recorded as a rollback to 1

kubectl -n tbtq get pods -l app=privatebin
# Healthy again
⚠️ “Deployed” does not mean “working.” By default helm upgrade returns as soon as the API server accepts the objects — it does not wait to see whether pods actually became ready, which is why the broken revision above reports success. Add --wait --timeout 5m to make Helm block until pods are ready, and --atomic to make it roll itself back automatically on failure. Worth putting both in the Phase 9 pipeline.
# The version you actually want in automation:
helm upgrade tbtq ./tbtq-chart -n tbtq --atomic --wait --timeout 5m

Alternative — adopting instead of rebuilding

If you’d rather not lose the data, you can hand Helm ownership of the existing objects by adding the labels and annotations it looks for. It only matters for objects the chart itself renders; annotating anything else does nothing. Include the PVCs here — unlike a StatefulSet’s auto-created volumes, the ones in this build are declared directly in the chart, so Helm should own them.
for kind in deployment service ingress pvc cronjob networkpolicy; do
  for obj in $(kubectl -n tbtq get $kind -o name 2>/dev/null); do
    kubectl -n tbtq annotate $obj \
      meta.helm.sh/release-name=tbtq \
      meta.helm.sh/release-namespace=tbtq --overwrite
    kubectl -n tbtq label $obj app.kubernetes.io/managed-by=Helm --overwrite
  done
done

# StatefulSet and its PVCs need care — adopt the StatefulSet only:
Then helm install as normal. If you ran kubectl autoscale in Phase 6, delete those HPAs first — they aren’t in the chart and will fight it.
✓ Chart renders clean with no unresolved values. Stack reinstalls from one command. Rollback exercised on a deliberate failure, not discovered during a real one.

Check yourself

If you can’t answer these, reread rather than moving on:
  • Why did helm upgrade report success on a broken image tag?
  • What would helm rollback have done to Send’s uploads, given the chart includes its PVC?
  • Where does {{ .Release.Namespace }} get its value, and what breaks if you hardcode tbtq instead?
  • Why does MAX_FILE_SIZE need | quote when replicas doesn’t?

Phase 9 — Cleanup + CI/CD Optional

Two unrelated things, take them separately. Neither is required for the stack to run, and they don’t depend on each other or on Phase 8.
  • Orphan cleanup — a safety net, not a requirement. Send deletes its own files on download or expiry; this only sweeps up blobs left behind by a crash mid-upload. Skipping it costs you disk very slowly.
  • CI/CD — convenience only. Editing manifests over SSH and running kubectl apply is fine for one box.
Why there’s no backup section. Both services are deliberately amnesiac. Send’s blobs are one-time downloads on a 24-hour timer; PrivateBin’s pastes burn on read or expire. Redis holds the metadata pointing at Send’s files, and already writes an append-only log to its volume, so a pod restart is survivable — which is the only failure worth surviving here. There is no accumulated state to protect, because the product is that nothing accumulates. Backing this up would mean preserving data both services promised to destroy.
The one thing worth copying, if you’re about to do something drastic to the node, is Redis’s append-only file. Lose it and any live Send link 404s while its blob sits uselessly on disk.
kubectl -n tbtq exec deploy/redis -- redis-cli BGREWRITEAOF
kubectl -n tbtq cp redis-<pod-suffix>:/data ./redis-snapshot

Orphan cleanup CronJob

What this is for: Send deletes its own files when the download count hits zero or the expiry timer fires. This job only catches blobs orphaned by a crash mid-upload, which Send itself will never revisit. alpine is a 5 MB base image — all this needs is find.
# manifests/cleanup.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: send-orphan-cleanup
  namespace: tbtq
spec:
  schedule: "17 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: send-orphan-cleanup
        spec:
          containers:
            - name: cleanup
              image: alpine:3.20
              command: ["/bin/sh", "-c"]
              args:
                - |
                  # MAX_EXPIRE_SECONDS is 86400, so anything past 48h is dead.
                  find /uploads -type f -mmin +2880 -print -delete
              volumeMounts:
                - name: send-data
                  mountPath: /uploads
          restartPolicy: OnFailure
          volumes:
            - name: send-data
              persistentVolumeClaim:
                claimName: send-data
Why 48 hours and not 24. Setting the threshold equal to the expiry window races Send’s own cleanup: this job could delete a blob while Redis still says the link is live, producing a 404 on a link the recipient was told is good. Doubling the window makes the two mechanisms non-overlapping. Note also that this job mounts the same ReadWriteOnce volume as the Send pod — fine on one node, but it will sit Pending if you ever add a second and the scheduler splits them.

GitHub Actions CI/CD

What’s running where: GitHub Actions runs a throwaway Ubuntu container on GitHub’s infrastructure. appleboy/ssh-action opens an SSH session from that runner to your VM using a private key stored in repository secrets, and runs the script there. Nothing is installed on your box beyond what’s already present.
# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to k3s
        uses: appleboy/ssh-action@v1
        with:
          host: ${{ secrets.VM_HOST }}
          username: root
          key: ${{ secrets.VM_SSH_KEY }}
          script: |
            set -euo pipefail
            cd ~/tbtq
            git pull --ff-only
            kubectl apply -f manifests/
            kubectl -n tbtq rollout status deployment/send       --timeout=120s
            kubectl -n tbtq rollout status deployment/privatebin --timeout=120s
This needs the repo cloned at ~/tbtq on the VM, and your gitignored secrets.yaml already sitting there. Note that apply is what makes this a deployment rather than a restart — rollout restart alone re-pulls images but never applies changed manifests, so you’d edit a resource limit, watch the pipeline go green, and the cluster would be unchanged.
On image tags. This guide uses moving tags (:latest, :stable, 16-alpine) because that’s what these projects publish. For anything you care about, pin to a digest — image: privatebin/nginx-fpm-alpine@sha256:... — and set imagePullPolicy: IfNotPresent. Otherwise “roll back the deployment” doesn’t roll back the code, and an unattended restart can quietly pull a new major version.
✓ Backups verified by restore. Orphan cleanup can’t race live links. Git push actually deploys.

Summary

Built for a box.

#PhaseWhat’s running
0VM + firewallUbuntu 24.04, ufw with cluster CIDRs allowed, API scoped to your IP
1k3s + HelmSingle node, Traefik with persistent certificate storage
2RedisSingle Deployment with AOF persistence on a 2Gi volume
3Send1 replica, Redis metadata, 20Gi blobs, 5GB per-file cap
4PrivateBin1 replica, zero-knowledge, fsGroup 82
4PrivateBin2 replicas + HPA, zero-knowledge, fsGroup 82
5Ingress + TLS2 subdomains, Traefik ACME, DNS verified first
6Observabilitymetrics-server + Traefik JSON access logs
6Observabilitymetrics-server, HPAs, Traefik logs, Prometheus + Grafana
7Network policiesDefault deny, explicit Redis path, tested both directions
— core build complete; everything below is optional —
8Helm OptPackaging practice. Requires rebuilding the namespace
9Cleanup + CI/CD OptOrphan sweep, apply-based deploys. No backups needed

Troubleshooting

SymptomCause
All pods stuck ContainerCreating, DNS failuresufw missing the pod/service CIDR or VXLAN rules — Phase 0
Send pod crashlooping on bootREDIS_HOST unset or Redis not ready — Phase 2/3
Browser shows TRAEFIK DEFAULT CERTACME challenge failed. Check DNS resolves and port 80 is open
Certs reissue on every restart, then stop workingACME persistence off, now rate-limited. Wait out the week on staging
PrivateBin 500s when saving a pasteVolume not writable — fsGroup: 82
Pods evicted, node under memory pressurePrometheus stack is the usual culprit — recheck the Phase 0 budget
Rollout hangs at ContainerCreating after adding a nodeReadWriteOnce volume contention. Recreate strategy, 1 replica
Public form returns 404 on POSTIngressRoute priority or the Method(`POST`) matcher — check kubectl -n tbtq describe ingressroute
GET on /api/create returns 200Method matcher not applied. Take the route down — listing is exposed
Domain flagged by Safe BrowsingShortener abused. Disable the public key, audit short-url:list, request review
Helm install fails on ownership metadataObjects were created by kubectl — rebuild the namespace or adopt them, Phase 8
helm upgrade succeeds but pods are brokenHelm doesn’t wait by default — add --atomic --wait
<no value> in rendered manifestsTemplate references a key missing from values.yaml — helm template catches it
Previous Article

Leave a Reply

Your email address will not be published. Required fields are marked *.

*
*