Failure Boundaries
Failure boundaries: the happy-path view vs how failure really flows, and eight key concepts for failure-resilient architecture: failure domains, blast-radius containment, bulkheads, dependency isolation, partial failure, cascading retries, backpressure and graceful degradation.
In this diagram
- THE ARCHITECTURE NOBODY DRAWS: FAILURE BOUNDARIES
- Most architecture diagrams show how data flows.
- Senior architects spend more time thinking about how failure flows.
- THE HAPPY-PATH VIEW
- User
- API Gateway
- Services
- Database
- Looks simple. Works great.
- THE REALITY: HOW FAILURE FLOWS
- API Gateway ⚠
- Services ⚠
- Database ⚠
- A small failure can become a system-wide outage.
- Dependency gets slow → Retries multiply → Queues fill up → Cascades to other services
- KEY CONCEPTS FOR FAILURE-RESILIENT ARCHITECTURE
- FAILURE DOMAINS vs DEPLOYMENT DOMAINS
- us-east-1 ✕ down
- us-west-2 ✓ healthy
- A failure in one domain should not take down another. Design boundaries intentionally.
- BLAST-RADIUS CONTAINMENT
- ✹ Failure
- Service
- ⊗ Isolated
- Limit the impact of failures to a small part of the system.
- BULKHEADS
- Payments
- Orders
- Catalog
- Isolate critical components so one failure doesn't sink everything.
- DEPENDENCY ISOLATION
- Your Service
- ⚠ 3rd Party API
- ⚠ Payment Provider
- ✓ Email Service
- Use timeouts, circuit breakers and fallbacks for external dependencies.
- PARTIAL FAILURE
- Search (OK)
- Recommend- ations (Degraded)
- Payments (Down)
- Not every part fails at once. Design for partial availability.
- CASCADING RETRIES
- Service A
- Service B
- Service C