The Case of the 79 Idle Connections

It started with a message from the DBA team. Our shared Postgres host was carrying 79 idle connections from a single service, ours, and they were capping our role at 20 until we sorted it out. By the time I read it, things were already worse: a database migration wouldn’t even start. FATAL: too many connections for role. Our own app pods had eaten the entire allowance, leaving nothing for the migration that was supposed to fix things. We were locked out of our own database by ourselves.

The first instinct is always “we have a leak somewhere.” We didn’t. What we had was an architecture quietly doing exactly what we told it to. Our Encore app runs about ten services, and every one of them was opening its own connection pool to the same database, each pool holding a minimum number of idle connections that sat around for minutes doing nothing. Ten pools per pod, times several pods. When I looked at what those idle connections last ran, it was always the same thing: an audit insert. Not a bug, just our fire-and-forget audit writer borrowing connections and politely handing them back, idle, over and over. Encore literally recommends one shared pool per process for exactly this reason. We’d drifted away from that, and nobody noticed, because on our own dev databases it never mattered. It only became a problem the moment it shared a server with everyone else.

That’s the part worth sitting with. The naive fixes all make it worse. Add more replicas? You multiply the pools. Raise the max connections? You multiply the ceiling. We kept reaching for tuning knobs when the real number was never “how many queries”, it was “how many pools open connections,” and that was quietly multiplying with every service and every pod we added.

The fix, once we saw it clearly, was almost embarrassingly small. We collapsed all the per-service pools into a single shared one per process, the pattern the framework wanted all along, set the idle floor to zero so a quiet pod holds no connections at all, and shortened how long idle connections linger. That one change took us from 79 idle connections down to 2. Along the way we untangled the chain that started the whole fire: an ingestion job spiking audit writes, which traced to unbounded concurrency, which traced to a cloud-storage upload buffer left at its 16 MB default, thirty-two of those at once was quietly OOM-killing pods and showing up, three layers away, as mysterious workflow timeouts.

What I took away from it: connection count isn’t a setting you tune, it’s a shape your architecture has. You fix it by removing pools, not shrinking them. A convenience that’s harmless in isolation can be destructive the moment it shares a resource. And symptoms are almost always lying about their cause, our “timeouts” were really out-of-memory kills, the OOM was really a storage default, and “too many connections” was really an idle-floor multiplier nobody had ever multiplied out on paper. Every one of those only got fixed when we stopped patching the surface and followed it down to the root. The actual fix was a few lines. Finding it was the whole job.