Implement a circuit breaker (Staff+ resilience extension)

Expected question

"Implement a circuit breaker with closed, open, and half-open states around an unreliable dependency."

Variant forms

"Implement a circuit breaker around an HTTP client."
"Model closed, open, and half-open transitions."
"Open after N consecutive failures, then retry after a cooldown."
"Make the breaker safe when many requests fail concurrently."
"Should failure rate or consecutive failures trip the breaker?"
"How do you limit half-open probe traffic?"
"Add metrics and alerts for breaker state changes."
"How does a circuit breaker differ from retries, timeouts, and bulkheads?"

The question, as it might actually be asked

Implement a call(operation) wrapper. The breaker starts closed, opens after a configured number of consecutive failures, rejects calls while open, and permits one probe after a recovery timeout. A successful probe closes it; a failed probe reopens it.

How you'd talk while coding

I'd ask which exceptions count as dependency failures, consecutive vs rate trip, who owns timeouts, how many half-open probes, and reject vs fallback. Brute is retry-everything — you hammer a dying dependency harder. Correct is an explicit state machine: closed → open after N failures → half-open after cooldown with one probe. Critical: lock only for state transitions, never while running user code. Tests: Nth failure opens; open fails fast; one probe after cooldown; success closes, failure reopens. Staff+ stop line: "Breaker ≠ retries ≠ timeouts ≠ bulkhead. Prefer per-process breakers for fast local protection. I won't centralize breaker state as another SPOF unless they ask."

Where this actually gets asked

Platform / SRE / payments coding. Turns resilience into a testable local state machine. Staff candidates know a breaker cuts cascading load — it doesn't replace deadlines, retries, or isolation.

Problem

Implement a circuit breaker that wraps an operation and transitions through CLOSED, OPEN, and HALF_OPEN safely under concurrent calls.

Clarifying questions you should ask first

Which exceptions count as dependency failures versus caller/input errors?
Is the trip policy consecutive failures, failure rate, or both?
Who enforces operation timeouts: this wrapper or the caller?
How many half-open probes may execute concurrently?
Should rejected calls raise, return a fallback, or enqueue work elsewhere?

Approach ladder

Step	Idea
Brute	Retry every failing call independently; dependency receives more load during failure
Correct	Explicit state machine with failure threshold, cooldown, and one half-open probe
Staff+	Lock state transitions only; emit state metrics, bound in-flight work with a bulkhead, and define distributed scope

Reference solution (Python)

from __future__ import annotations

from enum import Enum
from threading import RLock
from time import monotonic
from typing import Callable, TypeVar


T = TypeVar("T")


class CircuitOpenError(RuntimeError):
    pass


class _State(str, Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitBreaker:
    """Concurrent circuit breaker with one half-open probe at a time."""

    def __init__(self, failure_threshold: int = 3, recovery_seconds: float = 30.0) -> None:
        if failure_threshold <= 0 or recovery_seconds < 0:
            raise ValueError("invalid circuit-breaker configuration")
        self._failure_threshold = failure_threshold
        self._recovery_seconds = recovery_seconds
        self._state = _State.CLOSED
        self._failures = 0
        self._opened_at = 0.0
        self._probe_in_flight = False
        self._lock = RLock()

    def call(self, operation: Callable[[], T]) -> T:
        with self._lock:
            now = monotonic()
            if self._state is _State.OPEN:
                if now - self._opened_at < self._recovery_seconds:
                    raise CircuitOpenError("circuit is open")
                if self._probe_in_flight:
                    raise CircuitOpenError("half-open probe already in flight")
                self._state = _State.HALF_OPEN
                self._probe_in_flight = True
            elif self._state is _State.HALF_OPEN:
                raise CircuitOpenError("half-open probe already in flight")

        try:
            result = operation()
        except Exception:
            self._record_failure()
            raise
        else:
            self._record_success()
            return result

    def _record_failure(self) -> None:
        with self._lock:
            self._probe_in_flight = False
            self._failures += 1
            if self._state is _State.HALF_OPEN or self._failures >= self._failure_threshold:
                self._state = _State.OPEN
                self._opened_at = monotonic()

    def _record_success(self) -> None:
        with self._lock:
            self._state = _State.CLOSED
            self._failures = 0
            self._probe_in_flight = False

Complexity: O(1) state work per call; operation latency is outside the breaker. State is O(1).

Tests / edge cases

Successful calls keep the breaker closed and reset the consecutive-failure count.
The Nth consecutive failure opens the breaker; later calls fail fast before invoking the operation.
After the recovery period, exactly one caller can execute the half-open probe.
A successful probe closes/reset the breaker; a failed probe reopens it and restarts cooldown.
Concurrent callers do not all become probes or execute user code while the breaker is open.

Staff+ deep dive

Topic	Talking point
Metrics	Emit state transitions, rejected calls, probe outcome, dependency latency, and fallback rate with dependency labels
Concurrency	Never hold the state lock while running the dependency; that would serialize calls and can deadlock
Bulkhead	A breaker limits calls after failures; a semaphore/pool limits concurrent in-flight calls before failures cascade
Distributed scope	Prefer per-process breakers for fast local protection; central state can synchronize policy but adds a failure dependency

What's expected at each level

Mid-level: Identifies the three states and implements basic transitions.
Senior: Handles cooldown, reset behavior, and failure boundaries with tests.
Staff+: Limits half-open probes under concurrency, separates breaker/retry/timeout responsibilities, and names useful metrics.
Principal: Tunes policy by dependency and tenant, connects state to SLO/error-budget response, and designs coordinated but failure-tolerant rollout.

Follow-up questions to expect

"Failure rate window?" — Bounded recent outcomes + minimum volume before trip. Consecutive failures alone can lie under low traffic.
"Add retries?" — Only idempotent calls, with deadlines and jitter. Breaker sees the final dependency outcome.
"What's a bulkhead?" — Bound concurrency/queues per dependency so one slow downstream can't exhaust every worker.