TIL
Set max idle connections, not just max open
Under load we kept getting “bad connection” errors that came and went. The connection pool had a sane max open setting, so that was not it.
The problem was the max idle connections, left at the default. So as soon as a burst finished, the pool closed most connections, then opened fresh ones for the next burst. Constant open/close churn, and some of those reused connections were already dead by the time we grabbed them.
Setting a reasonable max idle (close to max open here) plus a connection max lifetime fixed it:
db.SetMaxOpenConns(25)
db.SetMaxIdleConns(25)
db.SetConnMaxLifetime(5 * time.Minute)
The lifetime matters too so connections do not live forever and hit server-side timeouts.