My Circuit Breaker Has Two States and Lives in Postgres

The textbook circuit breaker assumes your process will still be there in 15 seconds. On serverless it won't. What I built instead when my catalog APIs went down.

Every circuit breaker diagram on the internet looks the same. Three states, a few arrows, one service protecting itself from another service inside the same company.

Mine has two states, lives in a Postgres table, and is woken up by whoever happens to visit my website. I did not set out to be weird about this. Serverless made me.

I'm building Epilogd, a place to log the anime you watch and the games you play. I don't own any of that catalog data. Game metadata comes from RAWG, anime metadata comes from Jikan, and every search on my site quietly depends on two free APIs run by other people.

Then one evening RAWG started returning HTTP 522, and my game search went down with it.

So here's the plan: I'll walk through what a circuit breaker normally is, why the normal one doesn't survive on serverless, and what I ended up with. Real code, real SQL, and the parts I still haven't seen work in production.

The textbook version

A circuit breaker sits between your code and something that might be broken. If you haven't met one, Martin Fowler's write-up is the classic. It has three states:

• Closed: everything is fine, and calls go through.
• Open: too many failures, so calls fail immediately without even trying.
• Half-open: after a while, let a few real requests through as a test. If they pass, close. If they fail, open again.

The usual reason to want one is cascading failure. Service A calls service B. B gets slow. A's threads all pile up waiting on B, so A gets slow too, and whatever calls A goes next. Failing fast stops the collapse from spreading.

To be clear, I didn't invent anything by pointing one at a third-party API. People have wrapped external dependencies in breakers for years. What's different here is everything the textbook quietly assumes.

That last row is the one that caused all the trouble.

The bean supplier

Let's bring back Eve from the event loop post. She runs a coffee shop, and she buys beans from a supplier called RAWG.

One morning RAWG stops answering the phone. Eve is not an idiot, so:

1. She lets it ring for a few seconds, then calls a backup supplier so this customer still gets coffee.
2. After three unanswered calls in a row, she stops ringing RAWG and goes straight to the backup. Nobody should stand in line listening to a phone ring.
3. Every so often, she tries RAWG again to see if they're back.

Lovely. That's a circuit breaker with a fallback, and if Eve were one long-running process, we'd be done.

But I deploy on Vercel. So here's the twist:

Eve has amnesia. There are also several Eves, nobody knows how many, and any of them can be replaced by a brand-new Eve at any moment. Between customers, they might not exist at all.

Every piece of breaker advice that begins with "keep a counter in memory" just died.

How I got this wrong first

My first fix was a simple switch in memory: if RAWG fails, flip to IGDB and stay there for the life of this instance.

It had two problems, and they're the same problem:

• New instances knew nothing. Each fresh Eve had to personally discover the outage by waiting on a timeout, so some unlucky visitor paid for it. Again and again.
• Old instances never came back. A long-lived instance that flipped would stay on the backup long after RAWG recovered. There was no way back.

So I added a probe at startup to warn new instances. Better, but now every cold start was pinging an API, and instances still couldn't agree with each other about anything.

The real issue was that the state belonged to no one. It needed to live somewhere that outlives any single Eve. Conveniently, I already had one of those.

Moving the brain into Postgres

So Eve gets a whiteboard in the back room. Every Eve, old or new, reads the same board.

SELECT provider, state, consecutive_failures, consecutive_probe_successes,
       cooldown_until, next_probe_at, probe_lease_until
FROM catalog_breakers;

One row per primary provider: RAWG, whose fallback is IGDB, and Jikan, whose fallback is the MyAnimeList API. An absent row means closed, so a healthy provider costs nothing to store.

Why Postgres and not Redis? Because it was already there. I didn't want to add a whole new piece of infrastructure to protect me from infrastructure being down.

Now, the obvious fear: did I just add a database query to every request? "Don't hit the database from your circuit breaker" is good advice, so this took some care:

• Each instance keeps a copy of the state. Once the copy is 2 seconds old, the next request answers from it anyway and refreshes it after the response. Most visitors never wait on this read.
• Sometimes a request does wait. That's a brand-new instance that knows nothing, or a copy that's 30 seconds old, because by then it's too old to act on.
• A read gets 1 second, tops. Past that, the instance uses the last state it knew.
• Healthy traffic writes nothing. A success only writes if there's a failure count worth resetting.

The routing itself reads like the diagram, which I'm quite proud of:

result[key] = async (...args: unknown[]) => {
  if (!second) return first.apply(primary, args);
  if ((await breaker.route()) === "fallback") return second.apply(fallback, args);
 
  let value: unknown;
  try {
    value = await first.apply(primary, args);
  } catch (error) {
    const reason = classify(error);
    if (!reason) throw error;
    await breaker.reportFailure(reason);
    try {
      return await second.apply(fallback, args);
    } catch {
      throw error;
    }
  }
  await breaker.reportSuccess();
  return value;
};

Two details worth stealing. It's a decorator, so the rest of the app has no idea any of this exists. And if both providers fail on the same call, the caller sees the primary's error, so my API's status codes stay exactly what they were before.

Eight Eves, one marker

Multiple instances means races. Say eight requests all fail at the same moment, on different instances. A read-then-write would lose counts, so every change is a single SQL statement:

INSERT INTO catalog_breakers (provider, consecutive_failures, ...)
VALUES ($1, 1, ...)
ON CONFLICT (provider) DO UPDATE SET
  consecutive_failures = catalog_breakers.consecutive_failures + 1,
  state = CASE
    WHEN catalog_breakers.consecutive_failures + 1 >= $2 THEN 'OPEN'
    ELSE catalog_breakers.state
  END,
  ...

The row lock from ON CONFLICT lines them up. Eight failures count as eight, and the breaker opens exactly once.

All the timing uses the database's now(). Instances never compare their own clocks, because I don't trust eight amnesiacs to agree on what time it is.

Why half-open had to go

Here's the part that surprised me. I started with the three-state design, and ended up deleting a state.

Think about what half-open actually does: it picks a real user and makes them the test. If the primary is still down, that person waits out a 3-second timeout before the fallback answers. On an internal service that may be a fair trade, since failing fast is the point and you have nothing better to offer. But I have a perfectly good backup answering in well under a second! Why would I ever make someone wait to help me run an experiment?

So visitors never touch the primary during recovery. Only probes do:

                 request
                    |
        read state (the instance's copy)
                    |
       +------------+-------------+
       |                          |
     CLOSED                  OPEN or cooling down
       |                          |
   call primary               call fallback
    |       |                     |
    ok    failed              respond
    |       |                     |
 respond  count it,               | (OPEN, probe due)
          call fallback,          v
          respond         after the response:
                          one probe of the primary
                                  |
                       3 passes in a row: CLOSED
                       a failure: count back to 0

The state column only ever says CLOSED or OPEN. A probe is one small real request to the primary, and it takes three passes in a row, 15 seconds apart, to close. One lucky response from a struggling API shouldn't send all my traffic back to it.

Who wakes the probes?

Here's the fun bit. A textbook breaker has a timer: "after 15 seconds, test again." But there's no process to hold a timer. And Vercel's Hobby plan isn't going to run a cron job every 15 seconds for me.

So the probes are started by traffic. When a request sees an open breaker with a probe due, it schedules one using Next.js's after(), which runs work once the response has finished:

export function runAfterResponse(work: () => Promise<void>): boolean {
  try {
    after(() => work().catch(() => console.warn("Deferred work failed.")));
    return true;
  } catch {
    return false;
  }
}

The visitor gets their page and leaves, and then the instance checks on the primary. No visitor ever waits for it. The state refresh from earlier leaves the same way. And I'm not relying on an unawaited promise surviving the end of a response, which on serverless is a coin flip.

The catch is that many requests can notice a due probe at once, and I only want one of them to ring RAWG. So there's a lease, and again it's one statement:

UPDATE catalog_breakers SET
  probe_lease_until = now() + interval '10 seconds'
WHERE provider = $1
  AND state = 'OPEN'
  AND next_probe_at <= now()
  AND (probe_lease_until IS NULL OR probe_lease_until <= now())
  AND (cooldown_until IS NULL OR cooldown_until <= now())
RETURNING (EXTRACT(EPOCH FROM probe_lease_until) * 1000)::bigint::text AS lease;

The first caller takes the row lock and sets the lease. Everyone else then fails the condition and gets nothing back. If the winner's instance dies mid-probe, the lease just expires and the next request takes over, so a crash can't lock the breaker forever.

The lease's expiry doubles as its token. This one came out of reviewing my own code: a slow probe could finish after its lease expired and overwrite the result of a newer probe. Now finishProbe only applies if the token still matches, and a stale probe changes nothing.

Not every failure is the same failure

A boring breaker counts errors. But the errors mean different things:

The 401 one matters more than it looks. If my API key expires, opening the breaker would disguise a config bug as an outage. So it logs loudly instead, and never opens.

And a success that lands late can't close an open breaker. Only probes close it. Otherwise one slow response from before the outage could undo the whole thing.

Did it work?

I let a real outage test it, because Jikan was conveniently down when I tested this on a preview deployment. I didn't have to force anything:

• The breaker opened by itself after 3 failures.
• A request that had to wait on the primary's timeout took 4,526 ms. Once the breaker was open, requests went straight to the fallback in 550 to 650 ms.
• In the logs, the response finished at 14:16:32, and the failed probe landed about 3 seconds later. The visitor never felt it.
• Three full lease, probe, release cycles ran inside real after() calls.

As I write this, it's live in production, with both RAWG and Jikan open and their probes still failing. Search keeps working on the backups, and nobody using the site would know.

The honest part: both providers have stayed down, so I haven't yet watched a breaker close in production. Closing after three passes is covered by tests, not by a real recovery. I'll trust it fully the day I see that log line.

What this costs

This isn't free, and I wouldn't use it everywhere:

• No traffic, no probes. If nobody visits, the breaker stays open, and the first visitor after a quiet night starts recovery. Fine for me. Not fine if you need recovery while nobody's around.
• It depends on the database. If Postgres doesn't answer, instances use the last known state. With no known state, they try the primary, with fallback per request. My logs have already caught this once, when the database didn't answer a read in time.
• Some requests still pay for a read. A new instance waits on its first one, up to 1 second, on top of whatever the page was already doing. The breaker also shares its database connections with the rest of the app.
• Recovery takes about 45 seconds at best. Three probes, 15 seconds apart. I'm happy to stay on the backup a little longer than necessary.
• Other instances can lag. Usually by a little over 2 seconds, and on a quiet instance, up to 30 seconds' worth of old state. A few requests may still try the primary after the breaker opens. They fall back if it fails, so the cost is a little latency.
• The backup isn't the same. IGDB and RAWG don't describe games the same way, so results differ a bit during an outage. Availability, not equivalence.
• It wouldn't fit a hot internal service. This works because failures are rare and my traffic is modest. I wouldn't send every failure from thousands of requests per second through one Postgres row.

If you have long-running servers, a normal in-memory breaker is simpler, and you should use that.

The tiny mental model

A circuit breaker isn't really three states and some arrows. It answers three questions:

1. Should this request even try? That's the routing.
2. What happens when it can't? Mine has a real backup, so failing fast turns into rerouting.
3. Who finds out it's healthy again? With no long-running process, that became a lease, a table, and whichever visitor showed up.

Everything else came from taking away one assumption, that your process will still be there in 15 seconds. Take that away and half-open goes first, then the in-memory counter, then the timer. What's left is a Postgres row and a very forgetful barista who still gets your coffee out on time.

Thanks for reading. Go check what happens when your favorite API stops picking up. ☕