ai-automation-consultant / Failure Lab
Upstream 500 Without Circuit Breaker
A vendor outage floods retries and delays every workflow in the queue.
Category
reliability
Difficulty
standard
Last Run
none
Incident Brief
Diagnose the failure mode and propose a bounded recovery plan.
Broken State
The vendor API returns 500, and the workflow keeps immediate retries in the hot path.
Symptoms
- Queue latency spikes for unrelated jobs.
- Vendor status page shows elevated errors.
Trace
docs.extract status=500 error=upstream_unavailable retry.count=24 retry.strategy=immediate