Incident Response
Browse all articles, tutorials, and guides about Incident Response
Posts
Disk Full but du Disagrees: Finding Deleted Files That Still Use Space
When df reports a full filesystem but du cannot find the bytes, an unlinked log held open by a process is often the missing space. Find the holder, release space safely, and prevent the next midnight disk-full incident.
Who's On Call for the AI Agent? A Production Runbook
An agent that can call production tools is a service with an incident path, not a clever prompt. Assign an owner, page only for harm in progress, test a kill switch, scope its identity, and keep enough evidence to explain every action.
DevOps Interviews Now Test How You Decide. How to Practice Situational Questions
Fewer interviewers ask what a Kubernetes Service is. More of them describe a broken system and ask what you do first. Here is what they grade, five worked scenarios with weak and strong answers, and a way to practice making the call.
Deploy a Complete AI Incident Backend on One Neon Branch
Build and verify a disposable incident-search backend with Neon Auth, Object Storage, Functions, AI Gateway, and Lakebase Postgres—then remove the complete environment with one guarded command.
GitHub's 2.9B Monthly Commits: Anatomy of an Outage
GitHub's August 17 outage began with a missed sidecar limit and escalated through retry storms. Learn which reliability controls your platform needs next.
We Built an On-Call Agent in Mastra: Where It Won and Where It Would Not
Most agent tutorials stop at the happy path. We built a real on-call agent on Mastra, then killed the process with SIGKILL at the exact moment it rolled back a deploy. It recovered the run. It also rolled the deploy back a second time. Here is what durable execution actually guarantees, and the code that makes it safe.
11 Laws That Quietly Run Your Engineering Team
Parkinson, Goodhart, Brooks, Chesterton. Eleven old "laws" that were not written about software but explain your incidents, your estimates, and your org chart better than most engineering blog posts.
AI SRE Agents: What They Actually Fix, and What They Will Happily Break
AI SRE is now its own category, with every incident vendor shipping an agent that investigates and remediates on its own. Here is the honest split: where these agents genuinely earn their keep, where they are oversold, and the one risk nobody puts on the marketing page.
When One Data Center Room Got Hot: AWS US-EAST-1, Coinbase, and the DR Drill That Was Not
On May 7, 2026, cooling failed in a single hall of one US-EAST-1 data center. Coinbase, FanDuel, and CME Group went down for hours, and Coinbase publicly confirmed their backup systems did not work as expected. Here is what happened, the multi-AZ checklist that would have caught it, and the AWS Fault Injection Simulator commands to run the drill before the next thermal event.