Blog
Notes on DevOps, Kubernetes, self-hosted infrastructure and consulting.
October 17, 2026
The Quiet Cause of Sporadic 502s: Idle Connection Timeouts and Keepalive Ordering
Load balancers, proxies and application servers close idle connections on different clocks. Get the order wrong and random 502s appear after quiet periods. The...
Read more →
October 16, 2026
Why Published Docker Ports Bypass Your Firewall: UFW, the DOCKER-USER Chain and the Bind Address
UFW reports that every incoming connection is denied, yet a port published by Docker can still be open to the internet. The path a packet really takes, closing...
Read more →
October 15, 2026
The Get-or-Create Race: Duplicate Rows Under Concurrency, Unique Constraints and ON CONFLICT Traps
Check-then-insert is flawless for one request and quietly produces duplicates under parallel ones. Why application locks fall short, why a unique constraint is...
Read more →
October 14, 2026
Ransomware Resistant Backups: Object Lock, Versioning and Separating Writer from Deleter
A backup that production can delete is a backup an intruder can delete. The two object lock modes, append only identities, where the key lives, and how this com...
Read more →
October 13, 2026
Using Redis as a Data Store: Memory Policy, Persistence and Its Limits as a Queue
Redis arrives as a cache and ends up holding sessions and the job queue. Choosing maxmemory-policy, the cost of RDB versus AOF, what async replication loses, an...
Read more →
October 12, 2026
Finding Out What Actually Filled Up: inodes, Deleted but Open Files and Reserved Blocks
"Disk full" maps to at least three different conditions: blocks, inodes and space reserved for root. Same symptom, different remedy. How to measure before you d...
Read more →
October 11, 2026
PostgreSQL Autovacuum and Table Bloat: Dead Tuples, Transactions That Block Cleanup, and Per-Table Tuning
Default autovacuum thresholds let cleanup fall behind on large tables. How to measure bloat, find the old transactions that freeze cleanup, tune per table, get...
Read more →
October 10, 2026
Hardening Services with systemd: ProtectSystem, DynamicUser, System Call Filters and the Options That Break Things
Shrinking a service's attack surface from the unit file without reaching for containers: a starting template, measuring with systemd-analyze security, and the p...
Read more →
October 9, 2026
Cardinality Explosions in Prometheus: Choosing Labels, What a Series Costs and Stopping It at the Source
Every unique label combination is its own series, and one unbounded label can run Prometheus out of memory. Measuring series counts, the closed-set rule for lab...
Read more →
October 8, 2026
The Real Client IP Behind a Reverse Proxy: X-Forwarded-For, Trusted Proxies and Spoofed Headers
Behind a proxy, an application only learns the client address from a header. Reading X-Forwarded-For from the right, defining trusted proxies by address, nginx,...
Read more →
October 7, 2026
Build Once, Promote by Digest: Shipping the Exact Image You Tested
Rebuilding after the release tag means the bytes you tested are not the bytes you ship. Pushing candidates by digest, tagging that same digest once tests pass,...
Read more →
October 6, 2026
Time-Dependent Flaky Tests: Wall Clocks, Failures Under Load, Second Rounding and Fake Clocks
Why tests tied to the wall clock (fixed sleeps, short timeouts, duration assertions, single-instant "not yet" checks) fail precisely under load, how to measure...
Read more →
October 5, 2026
Publishing Packages Without Long-Lived Tokens: Trusted Publishing with OIDC on npm, PyPI and crates.io
A publish token sitting in a CI secret leaks, expires and is broader than it needs to be. Trusted publishing swaps it for the CI system's OIDC identity and a to...
Read more →
October 4, 2026
PodDisruptionBudgets and Node Drains in Kubernetes: Writing the Budget and Why Maintenance Gets Stuck
A PodDisruptionBudget only limits voluntary disruptions, and a badly written one either protects nothing or blocks node drains forever. Choosing between minAvai...
Read more →
October 3, 2026
Zero-Downtime Secret Rotation: Dual Keys, Alternating Users and the Consumers Left Behind
Rotation outages are caused by consumers, not by the value. Overlap windows, dual keys and alternating users, measuring the cutover, and the long-running proces...
Read more →
October 2, 2026
Point-in-Time Recovery in PostgreSQL: WAL Archiving, pgBackRest and Restore Traps
A nightly pg_dump means losing up to a day of data. How to rewind PostgreSQL to one second before a bad query with physical backups and a continuous WAL archive...
Read more →
October 1, 2026
The Transactional Outbox: Writing to a Database and a Queue Without Losing Events
Why commit-then-publish loses events, how to build an outbox table, whether to relay by polling or CDC, and how to avoid the sequence gap, duplicate delivery an...
Read more →
September 30, 2026
API Rate Limiting: Token Buckets, Choosing the Right Key and Distributed Counter Pitfalls
Why fixed windows let twice the limit through, why the IP address is a poor key, how INCR plus EXPIRE in Redis can block a client forever, and why you should an...
Read more →
September 29, 2026
DNS Resolution in Kubernetes: ndots, Search Domains, CoreDNS Caching and the Five-Second Stalls
Why a pod turns one external lookup into eight DNS queries, what lowering ndots breaks, when autopath and NodeLocal DNSCache are worth it, where the exact five-...
Read more →
September 28, 2026
PostgreSQL Connection Pooling: PgBouncer's Three Modes, the Prepared Statement Trap and Multi-Tenant Sizing
How to choose between session, transaction and statement modes, what breaks in transaction mode, why the prepared statement error only appears under load, and h...
Read more →
September 27, 2026
Custom Domains in SaaS: Ownership Verification, CNAME Targets and TLS Traps
Serving a SaaS from a customer's own domain is not a single CNAME record. Tenant routing, proof of ownership and certificate issuance are three separate problem...
Read more →
September 26, 2026
Kubernetes Resource Requests and Limits: CPU Throttling, OOMKilled and QoS Classes
A request is the scheduler's reservation, a limit is the kernel's quota. Why CPU gets paused while memory gets killed, how QoS classes decide eviction order, an...
Read more →
September 25, 2026
Zero-Downtime Schema Migrations: Expand and Contract, Locks and PostgreSQL Pitfalls
Rolling deploys are not enough; a schema change breaks the moment old and new versions run side by side. How to migrate PostgreSQL without downtime using expand...
Read more →
September 24, 2026
Stateful Resources Under Helm: Renames, resource-policy and CRD Pitfalls
Helm deletes a renamed PVC, restarts a StatefulSet on empty disks, regenerates passwords on every upgrade and never updates a CRD. Why data gets lost when state...
Read more →
September 23, 2026
Automated TLS Certificates with ACME: Validation Method, Rate Limits and State You Must Keep
Choosing between HTTP-01, DNS-01 and TLS-ALPN-01, where Let's Encrypt rate limits will catch you, the state you actually need to back up, and pitfalls like spli...
Read more →
September 22, 2026
Zero-Downtime Deployments in Kubernetes: Probes, preStop and Getting Graceful Shutdown in the Right Order
A rolling update alone does not give you zero downtime. Unless readiness and startup probes, a preStop delay, terminationGracePeriodSeconds, maxUnavailable and...
Read more →
September 21, 2026
Silent Failures in Bash Scripts: set -e, pipefail and Where Strict Mode Abandons You
set -euo pipefail does not make your script safe, it only gets you started. -e switching off inside function bodies, pipefail failing on SIGPIPE, -u ignoring em...
Read more →
September 20, 2026
Why Scheduled Jobs Break Quietly: Missed Runs, Overlap and Single Instance in cron, systemd Timers and Kubernetes CronJobs
Scheduled jobs break quietly in three ways: the missed run, the overlapping run and the failure nobody sees. How cron, systemd timers and Kubernetes CronJobs an...
Read more →
September 19, 2026
Enforcing Policy in Kubernetes: OPA Gatekeeper vs. Kyverno, and When to Use Which
RBAC decides who can act, not whether the object they submit is actually valid. Admission-based policy engines close that gap. We compare OPA Gatekeeper, Kyvern...
Read more →
September 18, 2026
Backups are not enough: run restore drills
A green backup job gives false confidence, and an untested backup is not a backup. We cover the silent traps we hit, valid but incomplete archives, stale locks,...
Read more →
September 17, 2026
Trust the output, not the model: a last line of defense for automated content
When we built an automated content pipeline, the hard part was never writing the text but making sure it could never leak internal details. Here is why telling...
Read more →
September 16, 2026
The PID 1 Problem in Containers: Zombie Processes and When You Need an Init System
The first process in a container is not an ordinary process as far as the kernel is concerned. Here is how that difference leads to zombie process buildup, how...
Read more →
September 15, 2026
Autoscaling Pods in Kubernetes: Choosing Between HPA, VPA, and KEDA
HPA scales replica count, VPA scales resource sizing, KEDA scales event-driven load. Using one in place of another is the root cause of most autoscaling complai...
Read more →
September 14, 2026
Webhook Design: Retries, Idempotency, and Signature Verification
Webhooks look simple until production traffic exposes three hard questions: delivery guarantees, duplicate events, and sender authenticity. This post covers ret...
Read more →
September 13, 2026
The Empty Map Trap When Talking to the Kubernetes API
When a Kubernetes map field is empty, whether JSON encodes it as an array or an object depends on your language, and getting it wrong silently rejects the whole...
Read more →
September 12, 2026
When You Actually Need a Service Mesh, and When It's Just Complexity
A decision guide to the real differences between Istio, Linkerd, and Cilium, what actually justifies adopting a service mesh, and the concrete pitfalls that sho...
Read more →
September 11, 2026
What You Gain and Lose With an Immutable Operating System
A look at the three main approaches to read-only, image-based operating systems: what they actually buy you, and the costs that only show up after adoption.
Read more →
September 10, 2026
Identity and Authorization in SaaS: Single Sign-On, Roles, and Tenant Boundaries
In a multi-tenant SaaS product, authentication, role, and tenant boundary are three separate axes. How to pick an identity model, how to match tenants during OI...
Read more →
September 9, 2026
How to Design a Client That Keeps Working Offline
Keeping an app from crashing when the network drops is one problem; making sure the user's changes survive that drop is a different one. We compare caching, opt...
Read more →
September 8, 2026
Cilium and eBPF: What Changes in Network Policy and Observability
Why iptables-based network policy runs into scaling walls in large clusters, how eBPF fixes that, and what Cilium's identity-based L7 policies and Hubble's side...
Read more →
September 7, 2026
k6 vs Locust for Load Testing: Which One to Choose and When
k6 and Locust are the two most common open source load testing tools. The real difference between them is not speed, it's the concurrency model, distributed exe...
Read more →
September 6, 2026
Cypress or Playwright for End-to-End Testing: Which One and When
The architectural difference between Cypress and Playwright decides far more than the authoring experience. What to know before choosing between the two, and th...
Read more →
September 5, 2026
Actually Measuring RPO and RTO Targets in Disaster Recovery
A green nightly backup job does not mean your RPO and RTO targets are actually met. Only a real, repeated restore drill produces the true number. Here is how to...
Read more →
September 4, 2026
Metrics, Logs, Traces, and Error Tracking: Why You Need All Four
Metrics, logs, traces, and error tracking do not measure the same thing. Each answers a different question at a different cost. A guide to when you need which,...
Read more →
September 3, 2026
Gateway API: What Changes After Ingress
Ingress has carried the same three or four fields for a decade, and everything else got bolted on through controller-specific annotations. Gateway API splits th...
Read more →
September 2, 2026
Queues and Async Job Processing: Where They Help, Where They Hurt
Pushing every slow operation into a queue does not remove complexity, it relocates it. A practical look at when async processing actually pays off, what deliver...
Read more →
September 1, 2026
Alerting Rules: "Something Broke" vs. "Users Are Affected"
Cause-based alerts (high CPU, full disk, failed job) wear out on-call engineers without answering whether users are actually affected. How to move to SLO-based...
Read more →
September 1, 2026
Secret Management in Kubernetes: Which Approach Fits Where
Kubernetes' built-in Secret object is not encryption. Here is how to choose between Sealed Secrets, External Secrets Operator, SOPS, and Vault's injector based...
Read more →
August 31, 2026
filex: a self-hosted file manager you can embed
By the third time we wrote a file listing UI just to show a client an S3 bucket, it was clear this would not scale. filex is that job done properly: one package...
Read more →
August 29, 2026
Self-hosted error tracking: choosing between Sentry and GlitchTip
Self-hosted Sentry wants twenty-odd services and 16 GB of RAM; GlitchTip does the same job in 512 MB. A comparison of licensing, footprint and feature gaps, plu...
Read more →
August 25, 2026
Designing a multi-tenant SaaS: isolation strategies and what they really cost
Database per tenant, schema per tenant and shared tables with a tenant column carry very different migration, backup, noisy neighbour and leak costs. Which one...
Read more →
August 19, 2026
How to Do GitOps, and What It Actually Buys You
Why the pull model differs from push-based deployment, where self-healing turns into a fight, why environments belong in directories rather than branches, how t...
Read more →
August 14, 2026
Terraform or OpenTofu: the license question and the real technical differences
What the 2023 license change actually broke, why OpenTofu was born under the Linux Foundation, and where the two CLIs genuinely diverge today: state encryption,...
Read more →
August 8, 2026
Reproducible service deployment with NixOS
What declarative configuration, atomic switching and flake pinning actually buy you, where NixOS beats Docker and conventional configuration management, which t...
Read more →
August 4, 2026
What an active-active Kubernetes architecture really takes
Active-active is not about splitting traffic, it is about writing consistently in two regions at once, and the real work ends up in the data layer. The cost of...
Read more →
July 28, 2026
Backing up a Kubernetes cluster with Velero and restoring it into another one
The real problem with Kubernetes backups is not etcd but cluster objects and persistent volume data being captured separately. How Velero is built, when to choo...
Read more →
July 21, 2026
Kubernetes from scratch on Talos Linux: operating a cluster without a shell
Talos Linux is an immutable, API managed Kubernetes OS with no shell and no SSH on the node. How machine config works, the bootstrap and upgrade flow, debugging...
Read more →