Most website outages don't look like outages. There is no smoke, no red banner, no angry mob in your inbox. The server keeps humming, the homepage may even load - and yet the thing your visitors actually came to do has been broken for days. Sometimes weeks.
This article is about why that happens, told partly through a failure we lived through ourselves.
A true story: 23 days of downtime nobody noticed
One of our own earlier projects - a small SaaS with a Rust backend and a PostgreSQL database - went down for 23 days. Not a dramatic crash. Something far more boring: during routine server maintenance, the database password was rotated. The application never got the new one.
Here is the uncomfortable part: everything looked fine.
- The server was up, CPU low, disk healthy.
- The process manager showed the service as
active (running)- the process hadn't crashed, it just failed every request that touched the database. - The landing page still rendered, because it was mostly static.
- No error tracker fired, because errors were "handled" and logged to a file nobody read.
Every sign-in attempt, every form submission, every API call quietly returned an error for over three weeks. We found out by accident, while doing unrelated cleanup on the server. The fix took two minutes. The outage took 23 days, because detection, not repair, was the missing piece.
Why outages are silent
1. You don't visit your own website. You built it; you have no reason to browse it daily. Your customers do - and most of them won't email you when something breaks. They leave. Studies on cart abandonment vary, but the pattern is consistent: users rarely report errors, they just bounce.
2. "Up" is measured at the wrong layer. A ping tells you the server answers ICMP. A port check tells you something listens on 443. A homepage check tells you your CDN can serve cached HTML. None of those prove that the application can reach its database, that sign-in works, or that checkout completes. Our 23-day outage would have passed a ping check every single minute of it.
3. Error pages that return 200. Frameworks love friendly error pages. If your "Something went wrong" page returns HTTP 200, a naive monitor sees success. The check that would have saved us needed to assert on either the status code of a database-backed endpoint or on content that only renders when things genuinely work.
4. Partial failures. DNS resolves but one record is stale. The certificate renewed but the web server never reloaded it. The site works over IPv4 and fails over IPv6. The main domain is fine and www is not. Each of these breaks the site for a fraction of visitors - which makes the silence even deeper, because your own connection may be in the lucky fraction.
Why they stay down
Silent failure explains the first hour. It doesn't explain the third week. Outages persist because of how alerting fails:
- No alerting at all. "It never went down before" is a monitoring strategy right up until it isn't.
- Alert fatigue. The opposite failure: a tool that cried wolf on every network blip got muted months ago. The one real alert lands in a channel nobody watches.
- Alerts to dead ends. Notifications go to an email that filters them to spam, or to a former colleague, or to a Slack workspace from two reorgs ago.
- The watcher died too. Cron-based self-checks running on the same server as the app share its fate. When the server has a bad day, both go quiet - and quiet reads as "fine".
- No recovery signal. If you only alert on failure, you learn to distrust silence. Alerting on recovery ("back up after 14 minutes") teaches you that silence plus a working dashboard means health.
What actually works
You don't need an SRE team. You need a few unglamorous habits:
1. Monitor a full-stack endpoint. Create a health check that touches the database (a cheap SELECT 1 behind /api/health) and monitor that, not just the homepage. Assert on the status code.
2. Check from outside. The monitor must live on different infrastructure than the thing it monitors. Anything running on the same box is a diary, not an alarm.
3. Check often enough to matter. A 5-minute interval means an outage can be 4 minutes and 59 seconds old before you can possibly know. For anything transactional, aim for a check every minute.
4. *Alert on down and on recovery, with the outage duration in the recovery message. That single habit builds trust in the system and gives you real data about your reliability.
5. Send a periodic "all clear". A weekly summary that says "3 sites, 100% uptime, 0 incidents" has a hidden benefit: when the summaries stop arriving, that itself is a signal.
6. Watch the boring expiries.* Certificates and domains fail on a calendar, not under load. Fourteen days of warning turns a 2 a.m. incident into a Tuesday task.
The lesson we paid 23 days for
Downtime is not a lightning strike; it's entropy. Passwords rotate, certs expire, disks fill, DNS drifts. The question is not whether something will quietly break - it's how many minutes will pass before you know.
Make sure the answer is minutes. Ours, once, was 33,000 of them.